1. BaSaMa & HiWi
    2.  

Targeted Synthetic LiDAR Data Generation and 3D Object De-tection for Construction Sites

Institute
Lehrstuhl für Fördertechnik Materialfluss Logistik (TUM-ED)
Type
Semester Thesis / Master's Thesis / Student Job /
Content
experimental /  
Description

Background

Reliable 3D perception is a prerequisite for autonomous mobile robots and construction machines operating in shared construction environments. However, real LiDAR datasets from construction sites are scarce, costly to annotate, and often provide insufficient coverage of rare but safety-relevant situations, such as partially occluded workers, machinery in unusual poses, sparse LiDAR returns at long range, or objects close to other machinery and infrastructure.

At our chair, a high-fidelity construction-site environment in NVIDIA Isaac Sim is already available. This thesis builds directly on this infrastructure to investigate how synthetic training data should be generated, not only how much data should be generated, to improve LiDAR-based 3D object detection in difficult construction-site scenarios.

Objective

The goal is to develop and experimentally evaluate a reproducible synthetic LiDAR data-generation pipeline for construction-site 3D object detection. The central research question is:

Under an equal synthetic-data budget, does targeted sampling of difficult and rare construction-site scenarios improve 3D object-detection performance compared with uniform randomization?

Two main training datasets of equal size will be generated: one using uniform randomization and one deliberately oversampling predefined difficult scenarios. CenterPoint will serve as the primary 3D detector. Performance will be evaluated separately under representative operating conditions and on deliberately difficult challenge scenarios.

The thesis focuses on LiDAR-based 3D perception. Camera-based or 2D detection is outside the scope.

Your tasks will include:

Scene, Class, and Dataset Specification:

Audit the existing Isaac Sim construction site and define a consistent dataset specification, including object classes, coordinate systems, oriented 3D bounding-box conventions and train/validation/test splits. The primary detection classes should cover discrete objects such as persons, vehicles/machines and discrete obstacles. Extended structures such as long walls and fences will primarily be treated as environmental geometry rather than conventional object-detection targets.

Define a reproducible scenario and difficulty taxonomy based on factors such as sensor range, LiDAR point count, visibility/occlusion, object proximity, unusual machinery poses and scene clutter. These criteria will be used to distinguish nominal from difficult scenarios before the final detector evaluation.

Synthetic Dataset Generation and Quality Assurance:

Extend the provided scene with a parameterized and reproducible randomization pipeline for object poses and counts, asset variants, ego/sensor poses, trajectories and scenario layouts. After an initial pilot, generate approximately 4,000–6,000 training LiDAR frames per sampling strategy, with identical data budgets for uniform and targeted sampling.

Export detector-ready point clouds, oriented 3D boxes, class labels and relevant scenario metadata. Implement automated and visual quality checks for box-point alignment, coordinate/yaw conventions, invalid annotations, dataset diversity and leakage-free scenario-level splits.

3D Detector Training and Evaluation:

Train CenterPoint using the provided MMDetection3D setup with fixed preprocessing, training budget and evaluation configurations. The detector configuration should remain identical between datasets so that the data-generation strategy is the principal experimental variable.

Evaluate both trained models on: 

  • a representative test set reflecting normal construction-site conditions; and

  • a challenge test set containing predefined difficult and safety-relevant scenarios.

Primary metrics include 3D Average Precision, BEV Average Precision and recall. Additional analysis should investigate performance by object class, range, LiDAR point count, occlusion/difficulty and scenario type, as well as characteristic false-positive and localization errors.

A predefined absolute AP threshold is not required as the thesis success criterion. The main scientific result is the controlled comparison between uniform and targeted synthetic-data generation and the resulting detector error characteristics.

Expected outcomes

  • Reproducible Isaac Sim synthetic LiDAR generation pipeline and dataset specification.

  • Validated uniform and targeted datasets with representative and challenge test sets.

  • Reproducible CenterPoint training configurations and checkpoints.

  • Quantitative comparison of the two data-generation strategies.

  • Difficulty-stratified detector error analysis and a documented error profile reusable in subsequent perception and navigation research.

Optional Extensions

If time permits, the work may be extended by:

  • evaluating a second detector such as PointPillars or SECOND;

  • studying dataset-size learning curves or a mixed uniform/targeted strategy;

  • increasing the dataset size beyond the core 4,000–6,000 frames per strategy; or

  • deriving a measured detector error profile for downstream navigation experiments.

Requirements

Profile

  • Strong interest in robotics, 3D computer vision, machine learning and simulation.

  • Background in Robotics, Computer Science, Mechanical Engineering, or a related field.

  • Solid Python programming skills and familiarity with PyTorch or a comparable machine-learning framework.

  • Experience with object detection or NVIDIA Isaac Sim is advantageous.

  • Structured and independent working style.

  • Good English skills.

Application

If you are interested, please send your application including your CV and academic transcript

via email.

Possible start
sofort
Contact
Yuan-Jen Huang, M.Sc.
Room: MW 0501
Phone: +49 (89) 289 - 15931
yuan-jen.huangtum.de
Announcement