1. Theses & HiWi Jobs
    2.  

Trajectory-Token-Adapter: Integrating Vehicle Dynamics into Vision-Language Models

Institute
Professur für autonome Fahrzeugsysteme
Type
Master's Thesis /
Content
experimental / theoretical /  
Description

You will conduct your research at AVS!

Vision-Language Models (VLMs) are increasingly used for autonomous driving, but they lack explicit understanding of vehicle dynamics. They can describe what they see but cannot reason about whether a trajectory is physically feasible or what forces are involved. This limits their usefulness for safety-critical planning applications.

In this Master's thesis, you will develop a novel Trajectory-Token-Adapter architecture that converts continuous vehicle dynamics signals (position, velocity, acceleration) into discrete tokens that can be processed by VLMs alongside vision and language inputs. This enables, for the first time, a direct integration of physical dynamics as a modality in foundation models.

Your tasks will include:

  • Designing and implementing a tokenization framework for vehicle dynamics data
  • Developing a textual encoding baseline for systematic comparison
  • Training dynamics-language alignment using driving datasets
  • Evaluating trajectory understanding and assessment capabilities
  • Analyzing which dynamics parameters provide the most value for different tasks

With your work you will actively contribute to a conference publication.

Requirements

Requirements:

  • Strong Python programming skills
  • Experience with deep learning frameworks (PyTorch)
  • Familiarity with transformer architectures and VLMs
  • Interest in autonomous driving and multimodal learning
  • Organized and independent work attitude

A plus:

  • Experience with model training and fine-tuning
  • Prior project work in robotics or autonomous systems
  • Experience in Formula Student or similar project-based teams
  • Understanding of vehicle dynamics

Your Benefits:

  • Cutting-edge research in foundation models for robotics
  • Young and dynamic team
  • Academic and professional support
  • Organized and structured thesis project
  • Direct contribution to a scientific publication at a top robotics conference
Tags
AVS Schaefer
Possible start
sofort
Contact
Finn Rasmus Schäfer
finn.schaefertum.de