Generative AI for Synthetic Data Augmentation in Long-Tail Scenarios

Institut

Professur für autonome Fahrzeugsysteme

Typ

Semesterarbeit / Masterarbeit /

Inhalt

experimentell / theoretisch / konstruktiv /  

Beschreibung

Join us in leveraging cutting-edge Generative AI (including Diffusion Models, GANs, NeRF, and 3D Gaussian Splatting) to create realistic synthetic data, specifically targeting rare, safety-critical "long-tail" scenarios that are under-represented in real-world autonomous driving datasets!

Are you passionate about using advanced AI to solve real-world safety challenges and overcome data limitations? Do you want to work on innovative ways to train and test autonomous vehicles in situations they rarely encounter but must handle flawlessly? This project offers a unique opportunity to apply the power of generative models to enhance the robustness and reliability of self-driving systems.

The core challenge in autonomous driving is not just collecting data, but obtaining diverse and effective examples of vulnerabilities. Generative AI provides a powerful solution, moving beyond simple data augmentation to a targeted, safety-critical engineering tool. This means guiding the generation process to intentionally create scenarios that expose specific weaknesses in AV policies, potentially using reinforcement learning or adversarial methods alongside an LLM-driven reward function for "vulnerability discovery."

You will concentrate on generating challenging scenarios such as complex multi-agent interactions (e.g., multiple vehicles and pedestrians in a busy intersection), highly adverse weather conditions (e.g., heavy rain, dense fog), or unusual object behaviors (e.g., unexpected sudden stops, erratic lane changes). The project will involve developing sophisticated methods for controllable generation, ensuring that the synthetic data is not only diverse but also highly relevant and tailored to improve AV robustness. Rigorous evaluation will involve testing VLM/VLA models trained with your augmented synthetic data, comparing their generalization capabilities against models trained solely on real-world data.

 

Example Thesis Topics

• Diffusion Models for Adverse Weather Scene Generation: Develop and fine-tune diffusion models to generate high-fidelity synthetic driving scenes under extreme and varied adverse weather conditions (e.g., blizzard, dust storm, intense glare) for perception model training.

• Controllable GANs for Multi-Agent Interaction Synthesis: Implement and evaluate conditional GANs that can synthesize diverse and complex multi-agent interactions (e.g., specific cut-ins, coordinated pedestrian groups, emergency vehicle maneuvers) by controlling agents' behaviors and trajectories.

• NeRF/3DGS for Realistic Long-Tail Event Reconstruction: Utilize Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS) to reconstruct and render highly realistic 3D scenes of rare accidents or anomalous events, enabling diverse viewpoint and sensor data generation.

• Adversarial Synthetic Data Generation for Policy Testing: Design an adversarial framework where a generative model produces challenging scenarios specifically designed to "break" an existing AV policy, thereby identifying its vulnerabilities.

• LLM-Guided Scenario Generation for Vulnerability Discovery: Investigate using LLMs to define and guide the generation of novel, safety-critical scenarios by describing rare events or potential failure modes, which a generative model then synthesizes.

 

Technologies Used

Python, PyTorch/TensorFlow, Generative AI (Diffusion Models, GANs, NeRF, 3DGS), Autonomous Driving, Data Augmentation, Synthetic Data Generation, Computer Vision, 3D Reconstruction, Deep Learning, Simulation (e.g., CARLA, Scenic, Unity, Unreal Engine), Reinforcement Learning (optional), Large Language Models (LLMs - optional for guidance/reward functions), Data Pipelines.

Voraussetzungen

We're looking for students with a strong background in deep learning and a passion for tackling fundamental challenges in autonomous systems.

• Solid understanding of deep learning frameworks (PyTorch/TensorFlow).

• Experience with generative models (GANs, Diffusion Models, NeRF, 3DGS, VAEs).

• Familiarity with computer vision and 3D graphics concepts.

• Proficiency in Python.

• Motivation to work on data-driven solutions for real-world safety.

 

If you're ready to make a tangible impact on the future of autonomous vehicles, send us an initiative application.

Please include:

• A short motivation letter highlighting your interest in generative AI, data augmentation, and autonomous driving.

• Your CV.

• A recent transcript of records.

• (Optional) Any relevant project work or code samples demonstrating your experience in relevant fields.

Möglicher Beginn

sofort

Kontakt

Roberto Brusnicki
roberto.brusnickitum.de