World Models for Sim-to-Real Transfer and Safety Verification

Institut

Professur für autonome Fahrzeugsysteme

Typ

Semesterarbeit / Masterarbeit /

Inhalt

experimentell / theoretisch / konstruktiv /  

Beschreibung

Join us in developing and applying "World Models" – generative AI systems capable of predicting future environmental states – to revolutionize autonomous driving, significantly improving simulation-to-reality transfer and enabling robust safety verification!

Are you fascinated by how AI can anticipate the future and enhance decision-making in complex, dynamic environments? Do you want to work on a foundational technology that allows autonomous vehicles to "think ahead" and proactively ensure safety? This project offers a unique opportunity to push the boundaries of embodied AI and predictive intelligence for self-driving cars.

World models are rapidly emerging as a pivotal technology for autonomous driving, providing high-fidelity representations of dynamic environments and enhancing the modeling of scene evolution. The power of world models lies in enabling the autonomous vehicle to "imagine" and evaluate various future possibilities. This capability can be further amplified by integrating the reasoning abilities of LLMs to interpret and critically evaluate these predicted future states, especially for safety verification. Imagine an LLM analyzing a predicted scenario and identifying potential safety violations or sub-optimal outcomes, providing natural language feedback to refine the planning module. This creates a "reasoning over predictions" loop, adding a crucial layer of meta-cognition to the autonomous system.

You will investigate how Vision-Language Models (VLMs) and Vision-Language-Action (VLAs) can be integrated with, or form the core components of, a comprehensive world model. Your work will enable the system to predict future states of the driving environment, including the anticipated behaviors of other road agents. This capability is essential for robust planning, allowing the autonomous vehicle to select the safest and most optimal trajectory by evaluating various future outcomes. Your research will directly address the critical sim-to-real gap, which is crucial for rapid development and deployment, and significantly enhance proactive safety measures in autonomous vehicles.

 

Example Thesis Topics

• VLM-Powered Prediction of Multi-Agent Interactions: Develop a world model that leverages VLMs to accurately predict the future trajectories and behaviors of multiple interacting road agents (vehicles, pedestrians, cyclists) in complex scenarios like busy intersections.

• Generative World Models for 4D Occupancy Grid Forecasting: Research and implement a generative world model capable of forecasting detailed 4D (spatial and temporal) occupancy grids of dynamic environments, providing dense future predictions for path planning.

• LLM-Guided Safety Analysis of Predicted Futures: Integrate an LLM with a world model to analyze predicted future driving scenarios, identifying potential safety violations or undesirable outcomes, and providing natural language feedback for planning refinement.

• Sim-to-Real Transfer using Latent Space World Models: Explore how world models learned in simulation can be effectively transferred to real-world autonomous driving by aligning latent representations or through domain adaptation techniques.

• Anticipatory Planning with Uncertainty-Aware World Models: Develop planning algorithms that utilize world model predictions to evaluate future risks and optimize trajectories, explicitly incorporating and propagating prediction uncertainty to enhance proactive safety.

 

Technologies Used

Python, PyTorch/TensorFlow, Autonomous Driving, World Models, Generative Models (VAEs, GANs, Diffusion Models), Vision-Language Models (VLMs), Vision-Language-Action Models (VLAs), Predictive Modeling, Reinforcement Learning, Planning Algorithms, Simulation (e.g., CARLA, Waymo Open Dataset, nuScenes, HighwayEnv), Sensor Fusion, Large Language Models (LLMs - for reasoning/analysis).

 

---

Voraussetzungen

We're looking for students with a strong background in deep learning and a passion for tackling fundamental challenges in autonomous systems.

• Solid understanding of deep learning frameworks (PyTorch/TensorFlow).

• Experience with generative models (e.g., VAEs, GANs, Diffusion Models) or sequence modeling.

• Familiarity with computer vision, robotics, or reinforcement learning concepts.

• Proficiency in Python.

• Motivation to work on predictive intelligence for real-world safety.

 

If you're ready to make a tangible impact on the future of autonomous vehicles, send us an initiative application.

Please include:

• A short motivation letter highlighting your interest in world models, predictive AI, and autonomous driving.

• Your CV.

• A recent transcript of records.

• (Optional) Any relevant project work or code samples demonstrating your experience in relevant fields.

Möglicher Beginn

sofort

Kontakt

Roberto Brusnicki
roberto.brusnickitum.de