SpatialLM: Training Large Language Models for Structured Indoor Modeling
SpatialLM is a large language model designed to process 3D point cloud data
and generate structured 3D scene understanding outputs. These outputs include
architectural elements like walls, doors, windows, and oriented object boxes
with their semantic categories. Unlike previous methods which exploit
task-specific network designs, our model adheres to the standard multimodal LLM
architecture and is fine-tuned directly from open-source LLMs.
  To train SpatialLM, we collect a large-scale, high-quality synthetic dataset
consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with
ground-truth 3D annotations, and conduct a careful study on various modeling
and training decisions. On public benchmarks, our model gives state-of-the-art
performance in layout estimation and competitive results in 3D object
detection. With that, we show a feasible path for enhancing the spatial
understanding capabilities of modern LLMs for applications in augmented
reality, embodied robotics, and more.
684799083ec10bdd8ab4de8a
Yongsen Mao
claimed_verified
2025-06-10T08:43:08.018Z
63efbb1efc92a63ac81126d0
https://cdn-avatars.huggingface.co/v1/production/uploads/1676655314726-noauth.jpeg
Yongsen Mao
user
ysmao
684799083ec10bdd8ab4de8b
Junhao Zhong
684799083ec10bdd8ab4de8c
Chuan Fang
684799083ec10bdd8ab4de8d
Jia Zheng
claimed_verified
2025-06-10T08:43:11.939Z
6437c0ead38ce48bdd4b0067
https://cdn-avatars.huggingface.co/v1/production/uploads/6437c0ead38ce48bdd4b0067/9HdcjbD0ugiPypLDtQl8E.png
Jia Zheng
user
bertjiazheng
684799083ec10bdd8ab4de8e
Rui Tang
684799083ec10bdd8ab4de8f
Hao Zhu
684799083ec10bdd8ab4de90
Ping Tan
684799083ec10bdd8ab4de91
Zihan Zhou
SpatialLM: Training Large Language Models for Structured Indoor Modeling
SpatialLM is a large language model designed to process 3D point cloud data
and generate structured 3D scene understanding outputs. These outputs include
architectural elements like walls, doors, windows, and oriented object boxes
with their semantic categories. Unlike previous methods which exploit
task-specific network designs, our model adheres to the standard multimodal LLM
architecture and is fine-tuned directly from open-source LLMs.
  To train SpatialLM, we collect a large-scale, high-quality synthetic dataset
consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with
ground-truth 3D annotations, and conduct a careful study on various modeling
and training decisions. On public benchmarks, our model gives state-of-the-art
performance in layout estimation and competitive results in 3D object
detection. With that, we show a feasible path for enhancing the spatial
understanding capabilities of modern LLMs for applications in augmented
reality, embodied robotics, and more.
SpatialLM, a multimodal large language model, processes 3D point cloud data to generate structured scene understanding outputs, achieving state-of-the-art performance in layout estimation and competitive results in 3D object detection.
large language model 3D point cloud structured 3D scene understanding multimodal LLM fine-tuning synthetic dataset ground-truth 3D annotations layout estimation 3D object detection augmented reality embodied robotics