MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Multimodal large language models (MLLMs) excel at 2D visual understanding but
remain limited in their ability to reason about 3D space. In this work, we
leverage large-scale high-quality 3D scene data with open-set annotations to
introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation
benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data
covers diverse spatial tasks including spatial relationship prediction, metric
size and distance estimation, and 3D grounding. We show that CA-VQA enables us
to train MM-Spatial, a strong generalist MLLM that also achieves
state-of-the-art performance on 3D spatial understanding benchmarks, including
our own. We show how incorporating metric depth and multi-view inputs (provided
in CA-VQA) can further improve 3D understanding, and demonstrate that data
alone allows our model to achieve depth perception capabilities comparable to
dedicated monocular depth estimation models. We will publish our SFT dataset
and benchmark.
67da9cfc42f0b1995666d81d
Erik Daxberger
admin_assigned
2025-03-19T13:48:58.293Z
66150e514476535462a7166c
/avatars/75a0b6546dde9853a06c9536f067005d.svg
Erik Daxberger
user
edaxberger
67da9cfc42f0b1995666d81e
Nina Wenzel
admin_assigned
2025-03-19T13:49:04.691Z
65f9a3ebf8c8ab9966fccffc
/avatars/07e466225e1142d2fd7ebc2f23fe5062.svg
Nina Wenzel
user
nm-w
67da9cfc42f0b1995666d81f
David Griffiths
67da9cfc42f0b1995666d820
Haiming Gang
67da9cfc42f0b1995666d821
Justin Lazarow
admin_assigned
2025-03-19T13:49:38.680Z
66a13bb1c7a8aaa80e69ff00
/avatars/b09aff14b97dcbcfb7601e26bdc0dda2.svg
Justin Lazarow
user
jlazarow
67da9cfc42f0b1995666d822
Gefen Kohavi
admin_assigned
2025-03-19T13:49:44.261Z
6729c5cf10c844285570ccd8
/avatars/162a86583e71dcb005bf79d8e182da85.svg
Gefen Kohavi
user
gekoh
67da9cfc42f0b1995666d823
Kai Kang
67da9cfc42f0b1995666d824
Marcin Eichner
67da9cfc42f0b1995666d825
Yinfei Yang
admin_assigned
2025-03-19T13:49:54.995Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
67da9cfc42f0b1995666d826
Afshin Dehghan
admin_assigned
2025-03-19T13:50:01.179Z
66fc2377516eaf950d4b8209
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mcUxUxXy18Gv9KvCW23s0.png
Afshin Dehghan
user
afshindn
67da9cfc42f0b1995666d827
Peter Grasch
admin_assigned
2025-03-19T13:50:07.186Z
6649e0b6f50d471119285ab8
/avatars/1c93d0e15f2b68281a1a2ea052450a45.svg
Peter Grasch
user
petergrasch
MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Multimodal large language models (MLLMs) excel at 2D visual understanding but
remain limited in their ability to reason about 3D space. In this work, we
leverage large-scale high-quality 3D scene data with open-set annotations to
introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation
benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data
covers diverse spatial tasks including spatial relationship prediction, metric
size and distance estimation, and 3D grounding. We show that CA-VQA enables us
to train MM-Spatial, a strong generalist MLLM that also achieves
state-of-the-art performance on 3D spatial understanding benchmarks, including
our own. We show how incorporating metric depth and multi-view inputs (provided
in CA-VQA) can further improve 3D understanding, and demonstrate that data
alone allows our model to achieve depth perception capabilities comparable to
dedicated monocular depth estimation models. We will publish our SFT dataset
and benchmark.
A new dataset and benchmark enable a multimodal large language model to improve 3D spatial understanding, including depth perception, through supervised fine-tuning with diverse spatial tasks.
multimodal large language models MLLMs 3D space supervised fine-tuning evaluation benchmark indoor scenes Cubify Anything VQA CA-VQA spatial relationship prediction metric size distance estimation 3D grounding MM-Spatial depth perception metric depth multi-view inputs