Autonomous Racing: Recurrent Convolutional Neural Network Based Interface between Sim-Racing Video Stream and Autonomous Driving Software

Institut

Lehrstuhl für Fahrzeugtechnik

Typ

Semesterarbeit / Masterarbeit /

Inhalt

experimentell / theoretisch /  

Beschreibung

Ever dreamed of turning your sim racing passion into cutting-edge research? Well, get ready, because this thesis
maybe your chance for academic glory!
Imagine using deep learning to decode the chaos of a virtual racing—transforming raw video stream into precise vehicle states, not just for your own car, but for every opponent
on screen. If that wasn’t cool enough, this project plugs directly into TUM Autonomous Motorsport’s software,
the very same stack that dominates real-world autonomous racing, winning both the Indy Autonomous Chal-
lenge (IAC) and Abu Dhabi Autonomous Racing League (A2RL)
. With this interface layer, we can finally pit our real world racing stack against both game AIs and human players, creating the ultimate simulation
battleground. Whether you're here for the thrill of competition, the challenge of computer vision, or just an excuse
to "study" sim racing all day, this project has something for you.
Your task is to develop a computer vision-based interface layer capable of translating video streams from sim rac-
ing games into accurate ego and opponent vehicle states. This includes extracting global position, heading, accel-
erations, and angular velocities. The neural network may also integrate force feedback signals from the games to
enhance the accuracy of state estimation. The resulting system will serve as a bridge between simulation and
real-world autonomous racing by enabling the TUM Autonomous Motorsport software stack to be tested across
different simulators. This will facilitate benchmarking against in-game AI and human players, allowing for compre-
hensive performance evaluation. You will have the flexibility to explore various neural network architectures, in-
cluding ConvRNN based end-to-end learning approaches, modular pipelines combining visual SLAM with offline
maps, and frame to frame based state estimation models, potentially combined with late fusion techniques.
Disclaimer regarding the scope of this topic: getting sufficient accuracy out of this approach may be difficult
and scientific literature on how to achieve the goal is scarce, if you are not ready to put in exceptional effort and
don’t bring the right knowledge and correct attitude to this topic, I would recommend you pick something else.

Voraussetzungen

• Enthusiasm about deep learning and autonomous racing and sim-racing
• Good programming skills in Python and C++
• Previous experience with deep learning frameworks is RECOMMENDED
• Ability to collaborate in a team and engage in interdisciplinary research

Tags

FTM Studienarbeit, FTM Automatisiertes Fahren, FTM Buettner, FTM Informatik

Möglicher Beginn

sofort

Kontakt

Sascha Büttner, M.Sc.
sascha.buettnertum.de

Ausschreibung