Project Database

This project has been taken for the 2026 Fall semester.

INTRODUCTION

Egocentric wearable sensing is becoming a key enabler for embodied AI and robotics. Meta’s Project Aria research glasses provide rich, multimodal, first-person observations, including RGB video, scene cameras, IMUs, microphones, eye-tracking, and more, in a socially acceptable, all-day wearable form factor. They are specifically designed to advance egocentric AI, robotics, and contextual perception research.

In collaboration with Meta, we aim to tightly couple Aria research glasses with our existing manipulation platforms at EPFL. This project will:

1. Integrate the Aria Research Kit, including Aria Gen 2, with our ROS2-based robot platforms, including ViperX 300S and WidowX-250 arms on a mobile base. This includes calibration and time-synchronisation with RGB-D cameras and robot state.
2. Design and execute egocentric data collection in household-like environments, combining Aria, RealSense cameras, robot joint trajectories, and language annotations.
3. Develop egocentric data-driven policies for robotic manipulation through learning from demonstration. The robot can be a robot arm or a humanoid robot such as Reaman.
4. Explore one or more robotics applications powered by Aria signals, such as intention-aware teleoperation, egocentric demonstrations for policy learning, or vision-language-action fine-tuning for assistance tasks.
5. Perform systematic platform testing, validation, and documentation to deliver a reusable research pipeline for future projects. Excellent programming skills in Python are a plus.

IMPORTANCE

We have well-documented tutorials on using the robots, teleoperation interfaces for data collection, using the HPC cluster, and a complete pipeline for training robot policies. The Aria Research Kit and tools, including recording, calibration, dataset tooling, and SDK, will be integrated into this ecosystem. This allows the student to focus on the research questions rather than low-level setup. We already have a good base and results from the ongoing project.

WHAT MAKES ARIA SPECIAL FOR ROBOTICS?

Project Aria glasses are multi-sensor research smart glasses. They include multiple cameras with wide field of view, IMUs, microphones, eye gaze, and a Machine Perception Service that provides SLAM poses, hand poses, and related perception outputs. They are explicitly positioned by Meta as a research kit for contextual AI and robotics, especially for using egocentric sensing to build embodied agents that understand and act in the world.

Compared to a normal RGB-D camera, Aria provides:

• Egocentric view: what the human or robot sees while acting.
• Calibrated head pose and trajectory through SLAM in MPS.
• Hand and gaze information, depending on which parts of the system are used.
• A portable, wearable, and socially acceptable form factor.

WHAT WE HAVE

1. Ready-and-easy-to-use robot platforms: ViperX 300S and WidowX-250 arms, configured with 4 RealSense D405 cameras, various grippers, and a mobile robot platform.
2. Egocentric sensing hardware: Meta Project Aria research glasses, including Aria Gen 2, with access to the Aria Research Kit and tooling for data recording and processing.
3. Computing resources: two desktop PCs with NVIDIA GPUs 5090 and 4090.

CANDIDATES

Interested students can apply by emailing sichao.liu@epfl.ch or lixuan.tang@epfl.ch.

Please attach your transcript and a short description of your past/current experience on related topics, such as robotics, computer vision, machine learning, AR, or egocentric perception.

The position is open until we have final candidates. Otherwise, the position will be closed.

RECOMMENDED READING

1. Kareer, Simar, et al. “EgoMimic: Scaling Imitation Learning via Egocentric Video.” 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025.
2. Liu, Vincent, et al. “EgoZero: Robot Learning from Smart Glasses.” arXiv preprint arXiv:2505.20290, 2025.
3. Punamiya, Ryan, et al. “EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World.” arXiv preprint arXiv:2604.07607, 2026.
4. Saroha, Abhishek, et al. “EgoFlow: Gradient-Guided Flow Matching for Egocentric 6DoF Object Motion Generation.” arXiv preprint arXiv:2604.01421, 2026.
5. Zheng, Ruijie, et al. “EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data.” arXiv preprint arXiv:2602.16710, 2026.
6. Aria Project: https://www.projectaria.com/resources/
7. Aria GitHub: https://github.com/facebookresearch/projectaria_tools

This project has been taken for the 2026 Fall semester.

Background

To improve the indoor autonomy of people with mobility impairments, including wheelchair users, everyday furniture can be augmented with mobility and manipulation capabilities. Such robotic furniture could reorganize room layouts, clear pathways, transport objects, and deliver everyday items to users.

This semester project focuses on the user-centric design of such an assistive environment. The goal is to understand the everyday challenges, functional limitations, and interaction needs of wheelchair users, and to investigate how multiple user interfaces can provide accessible and intuitive control of robotic furniture.

Current Progress

1. Mobile robotic furniture: A group of robotic furniture units capable of autonomous navigation and room reconfiguration.
   References: Publication 1, Publication 2


2. Markerless body-language interaction: A multi-view RGB-D system tracks users and furniture in real time and enables simple gesture-based interaction.
   References: AI for Good 2026 demonstration


3. LLM/VLM-supported natural-language interface: Users can give commands such as “Bring me a cup of water,” while the system interprets the request and identifies relevant objects and furniture.


4. Origami-inspired robotic arm: A compact foldable robotic arm can be integrated with robotic furniture or the wheelchair for grasping and delivery tasks.
   Reference: Origami-inspired third arm


5. Tablet interface: A bird's-eye visualization allows users to select predefined layouts or reposition furniture through drag-and-drop interaction.

Project Objectives

1. User-Centric Study: Investigate the everyday challenges faced by wheelchair users through literature review and existing user surveys. Understand how different mobility impairments affect functional abilities and interaction needs, identify meaningful assistive applications, and define requirements for future user studies.


2. Multimodal Interface Integration: Investigate how origami arm, gesture, tablet, natural language, and/or other interfaces can complement each other. The aim is to ensure that users with different abilities or under different conditions always have an accessible way to trigger, modify, or stop an assistive function.


3. Interface Exploration: Depending on the student's interests, further develop or evaluate one specific interaction modality.

Desired Candidate Profile

• Strong interest in user-centric research, assistive technology, rehabilitation, or human–robot interaction;
• Interest in understanding how mobility impairments affect functional abilities and interaction with technology;
• Background in neuroscience, neuroengineering, biomedical engineering, rehabilitation, robotics, HCI, or related fields is welcome;
• Experience with user studies, experimental design, rapid prototyping, or robotic systems is an advantage.

This project has been taken for 2026 Fall semester.

INTRODUCTION

Generalist Vision-Language-Action (VLA) policies can now perform a wide range of robot manipulation skills, but evaluating and improving them remains slow and expensive: rigorous evaluation requires hundreds of real-world rollouts, and systematic improvement demands additional corrective data with expert labels. Generative world models offer a scalable alternative by letting policies roll out inside imagination space, and recent work has shown that controllable, multi-view, action-conditioned world models can both rank policy performance without real-world execution and boost success rates by synthesising successful trajectories for supervised fine-tuning. In parallel, goal-conditioned formulations specify tasks through a target visual or language goal rather than a scalar reward, enabling reward-free online improvement through hindsight relabelling.

OBJECTIVES

This project will tightly couple these three directions on our existing manipulation platforms at EPFL. Concretely, we will:

1. Integrate a controllable multi-view world model with our ROS2-based robot platforms (ViperX 300S and WidowX-250 arms, optionally a humanoid such as Reaman), including time-synchronisation with RGB-D cameras and robot state.
2. Adapt the world model into a goal-conditioned predictor that, given a current observation and a visual or language goal, imagines temporally consistent multi-view futures.
3. Implement an iterative co-improvement loop in which a small batch of real-world policy rollouts is used to ground the world model on contact-rich, deformable-object tasks, and the grounded model in turn generates large-scale synthetic data.
4. Replace the explicit success-classifier reward model with hindsight goal relabelling and LoRA-based fine-tuning of a π0.5-class VLA policy, enabling reward-free autonomous improvement.
5. Perform systematic evaluation, ablation, and documentation to deliver a reusable goal-conditioned RL pipeline for future projects.

IMPORTANCE

We have well-documented tutorials on using the robots, teleoperation interfaces for data collection, the HPC cluster, and a complete pipeline for training robot policies. Open-source codebases for π0.5 (openpi), Ctrl-World, Act2Goal, and video diffusion will be integrated into this ecosystem, so the student can focus on the research questions rather than low-level setup.

We already have a strong base and results from an ongoing project: world model and RL for robot manipulation. Compared to standard language-conditioned VLAs trained only on demonstrations, this paradigm gives us:

• Imagination-based policy evaluation that closely tracks real-world performance rankings, with no physical rollouts.
• Targeted policy improvement on unseen objects, novel spatial layouts, and out-of-distribution instructions via synthetic successful trajectories.
• Physically grounded predictions of contact-rich and deformable-object interactions once the world model is fine-tuned on online rollout data.
• Reward-free online adaptation through hindsight relabelling.

WHAT WE HAVE

1. Ready-and-easy-to-use robot platforms: ViperX 300S and WidowX-250 arms configured with 4 RealSense D405 cameras, various grippers, and a mobile robot platform, fully compatible with the LeRobot V3.0 data format.
2. Pretrained models: access to π0.5 / π0-FAST policies, the Ctrl-World world-model checkpoint, and Stable Video Diffusion backbones.
3. Computing resources: two desktop PCs with NVIDIA GPUs 5090 and 4090, plus HPC cluster access for world-model fine-tuning.

CANDIDATES

Interested students can apply by emailing sichao.liu@epfl.ch or lixuan.tang@epfl.ch. Please attach your transcript and a short description of your past/current experience on related topics such as robotics, computer vision, reinforcement learning, generative models, and VLA.

The position is open until we have final candidates. Otherwise, the position will be closed.

RECOMMENDED READING

1. “Ctrl-World: A Controllable Generative World Model for Robot Manipulation.” ICLR 2026. arXiv:2510.10125.
2. “VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model.” arXiv preprint arXiv:2602.12063, 2026.
3. “Act2Goal: From World Model To General Goal-conditioned Policy.” arXiv preprint arXiv:2512.23541, 2025.
4. “LeRobot: An Open-Source Library for End-to-End Robot Learning.” arXiv preprint arXiv:2602.22818, 2026.
5. “π0.5: A Vision-Language-Action Model with Open-World Generalization.” arXiv preprint arXiv:2504.16054, 2025.
6. “Evaluating Robot Policies in a World Model.” arXiv preprint arXiv:2506.00613, 2025.

This project has been taken for 2026 Fall semester.

Background

To improve the indoor autonomy of people with mobility impairments, including wheelchair users, everyday furniture can be augmented with mobility and manipulation capabilities. Such robotic furniture could autonomously reorganize room layouts, clear pathways, transport objects, and grasp and deliver everyday items to users.

This master’s thesis aims to integrate mobile robotic furniture, an origami-inspired robotic arm, markerless perception, and language-based interfaces into a complete assistive mobile manipulation system.

Current Progress

1. Mobile robotic furniture: We have developed a group of mobile robotic furniture units that can autonomously navigate and reconfigure indoor environments.
   References: Publication 1, Publication 2


2. Multi-robot dense coordination: Our Velocity Potential Field Modulation (VPFM) algorithm guides multiple robotic furniture units toward their target poses while avoiding collisions and deadlocks.
   Reference: VPFM publication


3. Markerless tracking and body-language interaction: A multi-view RGB-D system tracks users and furniture units in real time. It provides visual feedback for multi-robot coordination and enables users to control robotic furniture through simple hand motions. The single-view version has been published, while the multi-view version is currently under review.
   References: Single-view system, AI for Good Global Summit 2026 demonstration


4. LLM/VLM supported language interfaces: A natural-language interface supported by a large language model and a vision-language model allows the user to request tasks such as “Bring me a cup of water.” The system can interpret the request, localize the desired object, and identify the furniture on which it is placed. The related paper is under preparation, and a demonstration can be presented during the interview.


5. Origami-inspired robotic arm: The project uses an origami-inspired arm developed by the EPFL Reconfigurable Robotics Lab. The arm can remain folded when not in use and extend for grasping tasks, making it suitable for integration with robotic furniture.
   Reference: Origami-inspired third arm

Thesis Objectives

1. Mechanical and electronic integration: Integrate the origami-inspired arm with a mobile robotic furniture platform. An initial configuration is available, but the student will be encouraged to improve the design to increase the workspace, reachability, and stability of the mobile manipulator.


2. Whole-body control: Develop a controller that coordinates the mobile base and robotic arm. The controller will use feedback from the markerless tracking system and the VLM-based object-grounding module to generate commands for positioning, navigation, grasping, and collision avoidance.


3. End-to-end demonstration: Implement the following scenario:
   • A wheelchair user says, “Bring me a cup of water.”
   • The system interprets the request and localizes the target cup on a static piece of furniture.
   • The system may ask the user for confirmation when the request is ambiguous.
   • A mobile table equipped with the origami arm navigates to the object and positions itself for manipulation.
   • The arm grasps the cup, and the mobile table brings it back to the user.

The final demonstration will integrate speech interaction, visual grounding, markerless tracking, mobile navigation, whole-body control, and robotic grasping.

Desired Candidate Profile

• Experience with ROS 2, C++, and Python;
• Experience with mechanical design or rapid prototyping;
• Interest in mobile manipulation, assistive robotics, robot perception, or human–robot interaction;
• Experience developing real-world robotic applications is an advantage.

This master thesis will be co-supervised by BioRob and RRL

• Contact in BioRob: Lixuan Tang (lixuan.tang@epfl.ch)
• Contact in RRL: Alexander Schuessler (alexander.schuessler@epfl.ch)

This project has been taken for 2026 fall semester

Background

To improve the indoor autonomy of people with mobility impairments, including wheelchair users, everyday furniture can be equipped with mobility and manipulation capabilities. Such robotic furniture could autonomously reorganize room layouts, clear pathways, transport objects, and grasp and deliver everyday items to users.

Coordinating multiple robotic furniture units in confined indoor environments is challenging because their physical shapes and dimensions cannot be neglected. Unlike point robots, furniture units must account for their full geometry when navigating through narrow spaces and interacting with one another.

Current Progress

Our previously developed algorithm, Velocity Potential Field Modulation (VPFM), has demonstrated strong performance in the dense coordination of polytopic robot swarms in 2D environments. Given the current pose and velocity of each robot, together with its target pose, VPFM generates collision-aware motion commands that guide all robots toward their assigned targets while reducing the risk of collisions and deadlocks.

More information is available in our IEEE Robotics and Automation Letters publication and the corresponding open-source repository:

Paper: IEEE RA-L publication

Project Objectives

1. Adapting VPFM to an indoor maze map with static obstacles.
   An ordinary home environment consists of multiple rooms separated by walls and doors, as well as static furniture pieces. To ensure that each mobile robotic furniture unit can arrive at any reachable position in the home environment, combining VPFM with a global path planner can help the swarm of mobile robotic furniture to overcome bottleneck regions and resolve the deadlock.


2. Conduct simulation-based validation.
   Implement and evaluate the proposed methods in simulation using representative dense indoor scenarios. The evaluation should investigate collision avoidance, deadlock resolution, convergence to target poses, computational efficiency, and the benefits of combining a global path planner with the local VPFM coordinator.