1. BaSaMa & HiWi
    2.  

[SA/IDP/MA] LLM/VLM-Assisted Adversarial Scenario Generation and Editing for Autonomous Driving Simulation

Institute
Lehrstuhl für Fahrzeugtechnik (TUM-ED)
Type
Semester Thesis / Master's Thesis /
Content
experimental / theoretical /  
Description

Motivation

Constructing safety-critical scenarios manually is highly labor-intensive and lacks scalability. To automate this process, this project proposes an intelligent scenario generation and editing framework leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs). 

Unlike purely text-based instructions, we aim to develop a framework that ingests structured environment representations, including HD maps and surrounding dynamic object information. The core objective is to programmatically transform normal driving scenarios into challenging adversarial situations.  

 

Requirements

Work Packages

In this project, you will build a unified pipeline that automatically interprets normal driving layouts and crafts complex, adversarial edge cases. The work packages are organized as follows: 

  • Literature Review: Investigating state-of-the-art LLM and VLM-based critical/adversarial scenario generation techniques. 

  • Adversarial Editing Engine & Action Generator: Developing the core reasoning engine that changes other vehicle trajectories and outputs safety-critical maneuvers. 

  • Adversarial Scenario Dataset: generating structured datasets of adversarial scenarios in formats like CommonRoad or OpenSCENARIO, establishing a standardized database of corner cases for automated benchmarking. 

  • Verification through CARLA/esmini Execution: Executing and validating the generated adversarial scenarios in dual simulation environments.  

 

What you should bring along?

  • Very good programming skills in Python and PyTorch.

  • Familiarity with foundation models (LLMs/VLMs) and tools (CARLA, esmini, CommonRoad, OpenScenario) is a plus.

  • High personal motivation and independent working style.

  • Very good language proficiency in English.

     

Recommended literature: 

 

Possibility for publication in case of excellent work.

 

If you are interested, please send me a grade sheet, your CV, short introduction (~5 sentences why this topic is interesting to you), and earliest possible date!

Tags
FTM Studienarbeit, FTM AV, FTM AV Perception, FTM Lim, FTM Informatik, FTM IDP
Possible start
sofort
Contact
Hojun Lim, M.Sc.
hojun.limtum.de
Announcement