Foundation-Model-Assisted 3D Auto-Labeling Pipeline for Missed and Unknown Objects in Autonomous Driving

Institut

Lehrstuhl für Fahrzeugtechnik (TUM-ED)

Typ

Semesterarbeit / Masterarbeit /

Inhalt

experimentell / theoretisch /  

Beschreibung

Motivation

With the advancement of autonomous driving, many autonomous driving (AD) stacks are aiming at SAE level 4. However, even level 4 AD systems still encounter corner cases (CCs) and will disengage once they cannot solve the situation. A major cause of these disengagements is perception deficit, where pre-trained perception models fail to detect critical elements in the scene—either by completely missing them (Missed objects) or because the objects fall outside their predefined training categories (Unknown objects). To tackle these specific CCs will be an essential problem in autonomous driving. Therefore, this work focuses on bridging this perception gap by developing an automated 3D pseudo-labeling pipeline to generate explicit 3D bounding boxes specifically targeted at these missed and unknown dynamic objects

Voraussetzungen

Work Packages

In the project, you will develop a framework for generating explicit 3D pseudo-labels targeted at missed and unknown objects using multi-sensor fusion and foundation models. Furthermore, you will investigate the accuracy of the generated 3D bounding boxes based on point cloud projection techniques. The project can be described with the following tasks:

• Literature review: Foundation models for unknown object detection, and auto-labeling. 
• Implementation: Exanding an existing pipeline (VESPA) to specifically handling objects that are not detected by conventional closed-set perception models. 
• Evaluation: Validating accuracy and efficacy of generated labels.

What you should bring along?

• Very good programming skills in Python and PyTorch. 
• Knowledge of Computer Vision and Deep Learning (Must), Foundation models such as VLMs, SAM (Desired). 
• High personal motivation and independent working style. 
• Very good language proficiency in English.

 

Possibility for publication in case of excellent work.

 

If you are interested, please send me a grade sheet, your CV, and short introduction (~5 sentences why this topic is interesting to you)!

Tags

FTM Studienarbeit, FTM AV, FTM AV Perception, FTM Lim, FTM Informatik, FTM IDP

Möglicher Beginn

sofort

Kontakt

Hojun Lim, M.Sc.
hojun.limtum.de

Ausschreibung