Development of a Unified Evaluation Framework for Autonomous Driving in CARLA

Institute

Professur für autonome Fahrzeugsysteme (TUM-ED)

Type

Semester Thesis / Master's Thesis /

Content

 

Description

Motivation

The validation and safety assurance of modern autonomous vehicle systems is a cornerstone in the development of highly automated driving functions (Level 4+). In this context, there is a fundamental trade-off between simulation and real-world testing: while physical tests offer high physical validity, they are capital-intensive, time-consuming, and limited in scalability. In contrast, simulation platforms such as CARLA enable reproducible testing of complex traffic scenarios at a comparatively low cost and with high scalability. Across both real-world environments and simulations, a critical question persists: what defines 'good' autonomous driving?

Simply "completing a route" is not enough to qualify a self-driving agent as capable. To truly assess an agent's performance, we need a standardized "Digital Referee" that evaluates driving behavior across multiple dimensions.

While many existing benchmarks focus solely on goal reaching or collision avoidance (Safety), a sophisticated evaluator must consider the human element (Comfort) and operational costs (Efficiency). This thesis aims to build a robust, metric-based evaluator that processes CARLA telemetry data to provide a holistic driving assessment.

Objectives

The goal of this thesis is to design and implement a standardized evaluation tool that interfaces with CARLA and calculates key performance indicators (KPIs) in three core areas: Safety, Comfort, and Efficiency.

Planned work packages include:

• Metric Definition: Researching and selecting state-of-the-art mathematical formulations for driving quality.
• System Integration: Developing a Python-based toolchain that extracts real-time telemetry from the CARLA Python API.
• Evaluator Implementation: Building the logic to transform raw data (coordinates, velocities, accelerations) into normalized scores.
• Scenario Benchmarking: Testing the evaluator against different AD agents across various weather and traffic conditions.
• Visualization: Creating a "Reporting Dashboard" that summarizes the agent’s performance at the end of a run.

Requirements

• Interest in the simulation or evaluation of autonomous systems.
• Strong Python programming skills.
• A structured and independent working style.
• Proficiency in German or English.
• Prior experience with CARLA, ROS, or autonomous software stacks is helpful but not mandatory.

Start Date: Immediate / As soon as possible.

How to Apply: If interested, please submit your application via email, including your CV, current transcript of records, and a brief cover letter explaining why you would like to work on this subject.

Tags

AVS Bank

Possible start

sofort

Contact

Christoph Bank
christoph.banktum.de