Semantic Scene Graph & Large Language Model (LLM)–Empowered Autonomous Medical Robotics

Institut

Chair for Computer Aided Medical Procedures & Augmented Reality

Typ

Semesterarbeit / Masterarbeit /

Inhalt

experimentell / theoretisch / konstruktiv /  

Beschreibung

Background and Motivation:

Recent works such as [1][2] show that medical image (ultrasound) contain rich, learnable semantic structures, and that multimodal models can extract meaningful anatomical cues even with limited data. However, these advances focus mainly on image-level understanding, while robotic assisted medical imaging requires actionable spatial reasoning—the ability to understand anatomical relationships, detect missing tissue structures, and decide how the medical imaging sensor should move. To close this gap, we aim to bring semantic modeling into real autonomous robotic assisted scanning by combining Semantic Scene Graphs and LLMs, enabling robots to interpret medical image context and autonomously guide the real-time scanning process.

 

Research Objectives:

• Based our previous work to build scene graph based semantic understanding for medical image interpretation.
• Integrate advanced LLM reasoning to infer missing anatomy and generate imaging device guidance in real-time.
• Develop a  framework for autonomous robotic assisted medical imaging.
• Validate the system on real robotic platforms with real patient/phantom medical image data.

Voraussetzungen

Prerequisitess:

• Interests in Robotics, Computer Vision, and Deep Learning.
• Excellent experience in Pytorch and Robotic Control (ROS1/2 and Ubuntu experience will be plus)
• Prior Knowledge about medical ultrasound (not necessary)

What We Offer:

You will gain hands-on experience with real Franka robotic arm, advanced medical imaging systems and computer vision sensors. The project is expected to start as soon as possible, and depending on the progress, there will be a strong opportunity to submit the work to the upcoming IROS 2026.

Application:

Interested applicants please send the brief CV, academic transcripts via email M.Sc. Xuesong Li (xuesong.li@tum.de)

References:

[1] Li, Xuesong, et al. "Semantic scene graph for ultrasound image explanation and scanning guidance." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2025.

[2] Jiang, Zhongliang, Septimiu E. Salcudean, and Nassir Navab. "Robotic ultrasound imaging: State-of-the-art and future perspectives." Medical image analysis 89 (2023): 102878.

Möglicher Beginn

sofort

Kontakt

Xuesong Li
Xuesong.Litum.de