Learning to Forecast the Future: Physics-Informed Latent BEV World Models for Vision-based End to End Autonomous Driving

Institut

Professur für autonome Fahrzeugsysteme

Typ

Semesterarbeit / Masterarbeit /

Inhalt

 

Beschreibung

Autonomous driving continues to evolve by combining powerful perception, prediction, and planning systems. A critical part of this ecosystem is the vehicle’s ability to forecast its environment — not just in raw sensor space, but in a compact and structured format such as Bird’s-Eye View (BEV). Accurate future prediction in BEV is essential for robust planning and safe interaction with other agents on the road.

At the AVS Lab, we are exploring physics-informed latent world models that can predict future BEV scenes based on current observations. Inspired by recent advances in diffusion models and transformers, we aim to build a system that generates realistic, temporally consistent future BEV frames while embedding physical constraints such as kinematics and agent intent.

A concrete example can be find here.

Project Objective:
This project investigates the use of generative models (e.g., Diffusion Transformer, Latent Auto-regressive Models) to predict future BEV representations over a short time horizon. Unlike purely data-driven models, we introduce physics-based priors (e.g., acceleration limits, yaw dynamics, map constraints) to improve realism and generalization. The focus lies on:

• Building or extending a latent BEV world model
• Integrating physics constraints during training or inference
• Evaluating model quality in terms of realism, consistency, and physical feasibility
• Exploring downstream planning integration or multi-agent forecasting

We offer:

• Cutting-edge research in generative models and autonomous systems
• Integration with an existing end-to-end driving stack
• Opportunity to publish in top-tier conferences or journals
• Work in English, thesis supervision included

Your qualifications:

• Strong background in deep learning (familiarity with Python and PyTorch)
• Understanding of vehicle dynamics or physics-based modeling
• Passion for autonomous driving and structured prediction tasks
• You are not afraid of mathematical proofs concerning diffusion.

Start date is flexible. If you're interested, please send:

1. A highlight performance description of your computer science (e.g. deep learning, coding) related courses (can be within campus or online)
   (200 words would be enough)
2. Academic transcript (optional for now, but may be required later)
3. CV (optional)

to dingrui.wangtum.de.

Möglicher Beginn

sofort

Kontakt

Dingrui Wang
dingrui.wangtum.de