"3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding"

Semantic graphs make LLMs spatially aware, improving 3D scene comprehension significantly.

3DGraphLLM creates learnable 3D scene graph representations for LLMs, enabling better understanding of spatial relationships between objects in 3D environments.

https://arxiv.org/abs/2412.18450

🤖 Original Problem:

→ Current methods for 3D scene understanding with LLMs don't effectively use semantic relationships between objects, limiting their ability to interpret complex spatial queries.

-----

🔧 Solution in this Paper:

→ 3DGraphLLM converts 3D point clouds into graph vertices representing objects.

→ It uses pre-trained encoders to extract features from both objects and their relationships.

→ The system projects these features into LLM's token embedding space.

→ It creates a flat sequence using k-nearest neighbor selection with minimum distance filtering.

-----

💡 Key Insights:

→ Semantic relationships between objects significantly improve LLM responses

→ Using only 2 nearest neighbors provides optimal balance between performance and resource usage

→ Pre-training on ground truth segmentation improves model robustness

-----

📊 Results:

→ Multi3DRefer: +5.8% F1@0.5 improvement

→ ScanRefer: +4.4% Acc@0.5 boost

→ Scan2Cap: CIDEr@0.5 increased by 5.8%

------

Are you into AI and LLMs❓ Join my daily AI newsletter. I will send you 7 emails a week analyzing the highest signal AI developments. ↓↓

🎉 https://rohanpaul.substack.com/