Project Database

This project is intended as a summer project only.

In this project, the student will work closely with other team members to develop data collection pipelines for experiments with an amphibious robot, and use these pipelines to collect and analyze experimental data. Specifically, the student will:

• Use a 4-camera system to track the 3D kinematics of the robot, ideally in real time. This includes calibrating camera intrinsics and extrinsics, labeling data and training a DeepLabCut network or applying marker-tracking libraries for offline tracking, and implementing real-time tracking nodes in ROS 2.
• Synchronize data collected from multiple sources, including cameras, force sensors, motors, and other onboard sensors.
• Visualize the collected data in MuJoCo viewer, RViz, Blender, or other 3D visualization tools.
• Assist with experimental data collection.

The student is expected to be familiar with programming in C/C++ and Python, ROS 2, and robot kinematics. Experience with Docker, Linux kernel development, communication protocols, and computer vision algorithms would be a plus.

Students interested in this project should send the following materials to the project assistant: (1) a resume, (2) a transcript showing relevant courses and grades, and (3) any additional materials that demonstrate their skills and project experience, such as videos, slides, code repositories, or previous project reports.

This project has been taken for 2026 Fall semester.

Background

To improve the indoor autonomy of people with mobility impairments, including wheelchair users, everyday furniture can be augmented with mobility and manipulation capabilities. Such robotic furniture could autonomously reorganize room layouts, clear pathways, transport objects, and grasp and deliver everyday items to users.

This master’s thesis aims to integrate mobile robotic furniture, an origami-inspired robotic arm, markerless perception, and language-based interfaces into a complete assistive mobile manipulation system.

Current Progress

1. Mobile robotic furniture: We have developed a group of mobile robotic furniture units that can autonomously navigate and reconfigure indoor environments.
   References: Publication 1, Publication 2


2. Multi-robot dense coordination: Our Velocity Potential Field Modulation (VPFM) algorithm guides multiple robotic furniture units toward their target poses while avoiding collisions and deadlocks.
   Reference: VPFM publication


3. Markerless tracking and body-language interaction: A multi-view RGB-D system tracks users and furniture units in real time. It provides visual feedback for multi-robot coordination and enables users to control robotic furniture through simple hand motions. The single-view version has been published, while the multi-view version is currently under review.
   References: Single-view system, AI for Good Global Summit 2026 demonstration


4. LLM/VLM supported language interfaces: A natural-language interface supported by a large language model and a vision-language model allows the user to request tasks such as “Bring me a cup of water.” The system can interpret the request, localize the desired object, and identify the furniture on which it is placed. The related paper is under preparation, and a demonstration can be presented during the interview.


5. Origami-inspired robotic arm: The project uses an origami-inspired arm developed by the EPFL Reconfigurable Robotics Lab. The arm can remain folded when not in use and extend for grasping tasks, making it suitable for integration with robotic furniture.
   Reference: Origami-inspired third arm

Thesis Objectives

1. Mechanical and electronic integration: Integrate the origami-inspired arm with a mobile robotic furniture platform. An initial configuration is available, but the student will be encouraged to improve the design to increase the workspace, reachability, and stability of the mobile manipulator.


2. Whole-body control: Develop a controller that coordinates the mobile base and robotic arm. The controller will use feedback from the markerless tracking system and the VLM-based object-grounding module to generate commands for positioning, navigation, grasping, and collision avoidance.


3. End-to-end demonstration: Implement the following scenario:
   • A wheelchair user says, “Bring me a cup of water.”
   • The system interprets the request and localizes the target cup on a static piece of furniture.
   • The system may ask the user for confirmation when the request is ambiguous.
   • A mobile table equipped with the origami arm navigates to the object and positions itself for manipulation.
   • The arm grasps the cup, and the mobile table brings it back to the user.

The final demonstration will integrate speech interaction, visual grounding, markerless tracking, mobile navigation, whole-body control, and robotic grasping.

Desired Candidate Profile

• Experience with ROS 2, C++, and Python;
• Experience with mechanical design or rapid prototyping;
• Interest in mobile manipulation, assistive robotics, robot perception, or human–robot interaction;
• Experience developing real-world robotic applications is an advantage.

This master thesis will be co-supervised by BioRob and RRL

• Contact in BioRob: Lixuan Tang (lixuan.tang@epfl.ch)
• Contact in RRL: Alexander Schuessler (alexander.schuessler@epfl.ch)