title: SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning
    Capabilities
summary: Understanding and reasoning about spatial relationships is a fundamental
  capability for Visual Question Answering (VQA) and robotics. While Vision
  Language Models (VLM) have demonstrated remarkable performance in certain VQA
  benchmarks, they still lack capabilities in 3D spatial reasoning, such as
  recognizing quantitative relationships of physical objects like distances or
  size differences. We hypothesize that VLMs' limited spatial reasoning
  capability is due to the lack of 3D spatial knowledge in training data and aim
  to solve this problem by training VLMs with Internet-scale spatial reasoning
  data. To this end, we present a system to facilitate this approach. We first
  develop an automatic 3D spatial VQA data generation framework that scales up to
  2 billion VQA examples on 10 million real-world images. We then investigate
  various factors in the training recipe, including data quality, training
  pipeline, and VLM architecture. Our work features the first internet-scale 3D
  spatial reasoning dataset in metric space. By training a VLM on such data, we
  significantly enhance its ability on both qualitative and quantitative spatial
  VQA. Finally, we demonstrate that this VLM unlocks novel downstream
  applications in chain-of-thought spatial reasoning and robotics due to its
  quantitative estimation capability. Project website:
  https://spatial-vlm.github.io/
paper_authors._id: 65af3b811f418c7449386e3e
paper_authors.name: Boyuan Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:55:48.530Z
paper_authors._id: 646935e05d70156639500beb
paper_authors.avatarUrl: /avatars/1373ef2a40145aa505a03bfbae37c95a.svg
paper_authors.fullname: Boyuan Chen
paper_authors.type: user
paper_authors.user: buoyancy99
paper_authors._id: 65af3b811f418c7449386e3f
paper_authors.name: Zhuo Xu
paper_authors._id: 65af3b811f418c7449386e40
paper_authors.name: Sean Kirmani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:56:12.316Z
paper_authors._id: 648c85f33ffc11989bddcce1
paper_authors.avatarUrl: /avatars/bdde73b95a36e32cb2975656cca46022.svg
paper_authors.fullname: Sean Kirmani
paper_authors.type: user
paper_authors.user: skirmani
paper_authors._id: 65af3b811f418c7449386e41
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:56:28.114Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_authors._id: 65af3b811f418c7449386e42
paper_authors.name: Danny Driess
paper_authors._id: 65af3b811f418c7449386e43
paper_authors.name: Pete Florence
paper_authors._id: 65af3b811f418c7449386e44
paper_authors.name: Dorsa Sadigh
paper_authors._id: 65af3b811f418c7449386e45
paper_authors.name: Leonidas Guibas
paper_authors._id: 65af3b811f418c7449386e46
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:09:00.672Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_title: SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning
    Capabilities
paper_summary: Understanding and reasoning about spatial relationships is a fundamental
  capability for Visual Question Answering (VQA) and robotics. While Vision
  Language Models (VLM) have demonstrated remarkable performance in certain VQA
  benchmarks, they still lack capabilities in 3D spatial reasoning, such as
  recognizing quantitative relationships of physical objects like distances or
  size differences. We hypothesize that VLMs' limited spatial reasoning
  capability is due to the lack of 3D spatial knowledge in training data and aim
  to solve this problem by training VLMs with Internet-scale spatial reasoning
  data. To this end, we present a system to facilitate this approach. We first
  develop an automatic 3D spatial VQA data generation framework that scales up to
  2 billion VQA examples on 10 million real-world images. We then investigate
  various factors in the training recipe, including data quality, training
  pipeline, and VLM architecture. Our work features the first internet-scale 3D
  spatial reasoning dataset in metric space. By training a VLM on such data, we
  significantly enhance its ability on both qualitative and quantitative spatial
  VQA. Finally, we demonstrate that this VLM unlocks novel downstream
  applications in chain-of-thought spatial reasoning and robotics due to its
  quantitative estimation capability. Project website:
  https://spatial-vlm.github.io/
paper_ai_summary: A system for training Vision Language Models with internet-scale 3D spatial reasoning data improves their spatial understanding and enables novel applications in quantitative spatial reasoning and robotics.
paper_ai_keywords: Vision Language Models VQA 3D spatial reasoning automatic 3D spatial VQA data generation internet-scale 3D spatial reasoning dataset metric space chain-of-thought spatial reasoning