1. BaSaMa & HiWi
    2.  

Development and Evaluation of a Scalable Whole-Body Data Collection Framework for Vision-Language-Action Learning on Quadrupedal Mobile Manipulators

Institut
Professur für autonome Fahrzeugsysteme (TUM-ED)
Typ
Masterarbeit /
Inhalt
experimentell / theoretisch /  
Beschreibung

Motivation

Quadrupedal mobile manipulators combine agile locomotion with dexterous manipulation, enabling robots to operate in complex and unstructured environments such as warehouses, retail stores, industrial facilities, and homes. While model-based Whole-Body Controllers (WBCs) provide robust and physically feasible control, they lack the semantic reasoning and generalization capabilities required for open-world tasks. Conversely, recent Vision-Language-Action (VLA) models have demonstrated remarkable success in learning general-purpose robotic skills from large-scale multimodal datasets.

However, existing robotic datasets primarily focus on either manipulation or locomotion, while comprehensive datasets capturing coordinated whole-body behaviors remain largely unavailable. Collecting synchronized demonstrations that combine visual observations, proprioceptive states, language instructions, and whole-body action commands is therefore one of the key bottlenecks for developing next-generation embodied AI systems.

This thesis aims to develop a scalable simulation-based data collection framework that automates the generation, recording, and management of multimodal demonstrations for quadrupedal mobile manipulators, providing the foundation for future whole-body VLA models.

Goal

Develop an end-to-end framework for scalable whole-body data collection targeting Vision-Language-Action learning on quadrupedal mobile manipulators.

Specifically, the thesis aims to:

  • Develop a modular data collection pipeline in NVIDIA Isaac Sim.
  • Integrate an existing Whole-Body Controller (WBC) to generate expert demonstrations.
  • Record synchronized multimodal data, including RGB-D images, proprioceptive states, language instructions, and whole-body operational-space action commands.
  • Design a standardized dataset format suitable for future VLA training.
  • Evaluate the generated dataset with respect to scalability, diversity, synchronization quality, and downstream usability.
  • Outstanding results may lead to a joint publication at a top-tier robotics conference such as ICRA or IROS.

Expected Deliverables

  • A scalable whole-body data collection framework in NVIDIA Isaac Sim.
  • A standardized multimodal dataset containing expert whole-body demonstrations.
  • Automated recording, replay, synchronization, and dataset export tools.
  • Documentation and training scripts for dataset generation.
  • An evaluation report assessing data quality, diversity, and scalability.

Required Skills

  • Excellent English or German proficiency.
  • Strong Python programming skills.
  • Experience with ROS 2 and NVIDIA Isaac Sim is beneficial.
  • Basic knowledge of robotics, control, and computer vision.
  • Interest in embodied AI, robot learning, and Vision-Language-Action models.

Experience with quadrupedal robots or machine learning is helpful but not mandatory.

Start

Work can begin immediately.

If you are interested in this topic, please first have a look at our recent position paper:

Then send a brief cover letter explaining why you are fascinated by this subject, along with your current transcript of records and CV to:

yuan_avs.gao@tum.de

Tags
AVS Gao
Möglicher Beginn
sofort
Kontakt
Yuan Gao
yuan_avs.gaotum.de