title: MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
summary: Multimodal large language models (MLLMs) excel at 2D visual understanding but
  remain limited in their ability to reason about 3D space. In this work, we
  leverage large-scale high-quality 3D scene data with open-set annotations to
  introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation
  benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data
  covers diverse spatial tasks including spatial relationship prediction, metric
  size and distance estimation, and 3D grounding. We show that CA-VQA enables us
  to train MM-Spatial, a strong generalist MLLM that also achieves
  state-of-the-art performance on 3D spatial understanding benchmarks, including
  our own. We show how incorporating metric depth and multi-view inputs (provided
  in CA-VQA) can further improve 3D understanding, and demonstrate that data
  alone allows our model to achieve depth perception capabilities comparable to
  dedicated monocular depth estimation models. We will publish our SFT dataset
  and benchmark.
paper_authors._id: 67da9cfc42f0b1995666d81d
paper_authors.name: Erik Daxberger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:48:58.293Z
paper_authors._id: 66150e514476535462a7166c
paper_authors.avatarUrl: /avatars/75a0b6546dde9853a06c9536f067005d.svg
paper_authors.fullname: Erik Daxberger
paper_authors.type: user
paper_authors.user: edaxberger
paper_authors._id: 67da9cfc42f0b1995666d81e
paper_authors.name: Nina Wenzel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:49:04.691Z
paper_authors._id: 65f9a3ebf8c8ab9966fccffc
paper_authors.avatarUrl: /avatars/07e466225e1142d2fd7ebc2f23fe5062.svg
paper_authors.fullname: Nina Wenzel
paper_authors.type: user
paper_authors.user: nm-w
paper_authors._id: 67da9cfc42f0b1995666d81f
paper_authors.name: David Griffiths
paper_authors._id: 67da9cfc42f0b1995666d820
paper_authors.name: Haiming Gang
paper_authors._id: 67da9cfc42f0b1995666d821
paper_authors.name: Justin Lazarow
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:49:38.680Z
paper_authors._id: 66a13bb1c7a8aaa80e69ff00
paper_authors.avatarUrl: /avatars/b09aff14b97dcbcfb7601e26bdc0dda2.svg
paper_authors.fullname: Justin Lazarow
paper_authors.type: user
paper_authors.user: jlazarow
paper_authors._id: 67da9cfc42f0b1995666d822
paper_authors.name: Gefen Kohavi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:49:44.261Z
paper_authors._id: 6729c5cf10c844285570ccd8
paper_authors.avatarUrl: /avatars/162a86583e71dcb005bf79d8e182da85.svg
paper_authors.fullname: Gefen Kohavi
paper_authors.type: user
paper_authors.user: gekoh
paper_authors._id: 67da9cfc42f0b1995666d823
paper_authors.name: Kai Kang
paper_authors._id: 67da9cfc42f0b1995666d824
paper_authors.name: Marcin Eichner
paper_authors._id: 67da9cfc42f0b1995666d825
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:49:54.995Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_authors._id: 67da9cfc42f0b1995666d826
paper_authors.name: Afshin Dehghan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:50:01.179Z
paper_authors._id: 66fc2377516eaf950d4b8209
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/mcUxUxXy18Gv9KvCW23s0.png
paper_authors.fullname: Afshin Dehghan
paper_authors.type: user
paper_authors.user: afshindn
paper_authors._id: 67da9cfc42f0b1995666d827
paper_authors.name: Peter Grasch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:50:07.186Z
paper_authors._id: 6649e0b6f50d471119285ab8
paper_authors.avatarUrl: /avatars/1c93d0e15f2b68281a1a2ea052450a45.svg
paper_authors.fullname: Peter Grasch
paper_authors.type: user
paper_authors.user: petergrasch
paper_title: MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
paper_summary: Multimodal large language models (MLLMs) excel at 2D visual understanding but
  remain limited in their ability to reason about 3D space. In this work, we
  leverage large-scale high-quality 3D scene data with open-set annotations to
  introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation
  benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data
  covers diverse spatial tasks including spatial relationship prediction, metric
  size and distance estimation, and 3D grounding. We show that CA-VQA enables us
  to train MM-Spatial, a strong generalist MLLM that also achieves
  state-of-the-art performance on 3D spatial understanding benchmarks, including
  our own. We show how incorporating metric depth and multi-view inputs (provided
  in CA-VQA) can further improve 3D understanding, and demonstrate that data
  alone allows our model to achieve depth perception capabilities comparable to
  dedicated monocular depth estimation models. We will publish our SFT dataset
  and benchmark.
paper_ai_summary: A new dataset and benchmark enable a multimodal large language model to improve 3D spatial understanding, including depth perception, through supervised fine-tuning with diverse spatial tasks.
paper_ai_keywords: multimodal large language models MLLMs 3D space supervised fine-tuning evaluation benchmark indoor scenes Cubify Anything VQA CA-VQA spatial relationship prediction metric size distance estimation 3D grounding MM-Spatial depth perception metric depth multi-view inputs