Talk2Drive: A Conversational Preference-Aligned Driving Framework Using Multi-Modal Large Language Models

Institut

Professur für autonome Fahrzeugsysteme

Typ

Bachelorarbeit / Semesterarbeit / Masterarbeit /

Inhalt

experimentell / theoretisch /  

Beschreibung

Background

The autonomy of vehicles has advanced rapidly in recent years, reaching a level where human intervention is barely or not at all required in certain controlled environments. Leading the way are manufacturers that now offer Level 3 autonomous vehicles depending on the system’s design. This progress relies heavily on the development and validation of highly reliable driving functions. Ensuring their safety and reliability requires extensive testing in diverse and challenging scenarios.

In parallel, the rise of Multi-Modal Large Language Models (MLLMs) has opened new opportunities for human-centered autonomy. MLLMs combine scene understanding with natural dialog, enabling drivers to express preferences (e.g., comfort vs. speed, lane choices, headway) in plain language. While recent works explore LLMs across perception, decision-making, control, and simulation, far less attention has been paid to aligning driving behavior with a driver’s explicitly stated expectations in real time. This creates an exciting research direction: building conversational agents that translate dialog + visual context into safe, principle-aware high-level control signals such as target speed, following distance, or lane-change intent.

Objective

The primary objective of this project is to develop a conversational, preference-aligned driving framework that (i) elicits and tracks driver expectations via natural dialog and (ii) outputs high-level, safety-checked control commands for the motion planner. We will evaluate alignment quality, safety, comfort, and robustness to phrasing variations.

We Offer

• A dynamic and future-oriented research environment

• Hands-on experience with a state-of-the-art software stack for autonomous driving (CARLA / MetaDrive)

• Opportunity to publish a scientific paper (based on merit)

• The thesis can be written in either English or German

Requirements (What You Should Bring)

• Initiative and a creative, problem-solving mindset

• Excellent English or German proficiency

• Advanced knowledge of Python and ROS2

• Prior experience with autonomous vehicles, simulation (CARLA/MetaDrive), or Large Language Models is an advantage

• Familiarity with common software development tools (e.g., Git, Ubuntu) is desirable

Work can begin immediately.
If you are interested in this topic, please first have a look at our recent survey paper: https://arxiv.org/abs/2506.11526
Then send an email with a brief cover letter explaining why you are fascinated by this subject, along with a current transcript of records and your resume, to: yuan_avs.gao@tum.de

Tags

AVS GAO

Möglicher Beginn

sofort

Kontakt

Yuan Gao
yuan_avs.gaotum.de