SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning
  Capabilities
Understanding and reasoning about spatial relationships is a fundamental
capability for Visual Question Answering (VQA) and robotics. While Vision
Language Models (VLM) have demonstrated remarkable performance in certain VQA
benchmarks, they still lack capabilities in 3D spatial reasoning, such as
recognizing quantitative relationships of physical objects like distances or
size differences. We hypothesize that VLMs' limited spatial reasoning
capability is due to the lack of 3D spatial knowledge in training data and aim
to solve this problem by training VLMs with Internet-scale spatial reasoning
data. To this end, we present a system to facilitate this approach. We first
develop an automatic 3D spatial VQA data generation framework that scales up to
2 billion VQA examples on 10 million real-world images. We then investigate
various factors in the training recipe, including data quality, training
pipeline, and VLM architecture. Our work features the first internet-scale 3D
spatial reasoning dataset in metric space. By training a VLM on such data, we
significantly enhance its ability on both qualitative and quantitative spatial
VQA. Finally, we demonstrate that this VLM unlocks novel downstream
applications in chain-of-thought spatial reasoning and robotics due to its
quantitative estimation capability. Project website:
https://spatial-vlm.github.io/
65af3b811f418c7449386e3e
Boyuan Chen
admin_assigned
2024-01-23T09:55:48.530Z
646935e05d70156639500beb
/avatars/1373ef2a40145aa505a03bfbae37c95a.svg
Boyuan Chen
user
buoyancy99
65af3b811f418c7449386e3f
Zhuo Xu
65af3b811f418c7449386e40
Sean Kirmani
admin_assigned
2024-01-23T09:56:12.316Z
648c85f33ffc11989bddcce1
/avatars/bdde73b95a36e32cb2975656cca46022.svg
Sean Kirmani
user
skirmani
65af3b811f418c7449386e41
Brian Ichter
admin_assigned
2024-01-23T09:56:28.114Z
633a26ab474cfeb1a864dc56
/avatars/cd40accd894fc78810fc0d5108f413e9.svg
Brian I
user
brianichter
65af3b811f418c7449386e42
Danny Driess
65af3b811f418c7449386e43
Pete Florence
65af3b811f418c7449386e44
Dorsa Sadigh
65af3b811f418c7449386e45
Leonidas Guibas
65af3b811f418c7449386e46
Fei Xia
claimed_verified
2024-01-23T08:09:00.672Z
62cfa9ee9b96f22525c36d47
/avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
Fei Xia
user
fxia22
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning
  Capabilities
Understanding and reasoning about spatial relationships is a fundamental
capability for Visual Question Answering (VQA) and robotics. While Vision
Language Models (VLM) have demonstrated remarkable performance in certain VQA
benchmarks, they still lack capabilities in 3D spatial reasoning, such as
recognizing quantitative relationships of physical objects like distances or
size differences. We hypothesize that VLMs' limited spatial reasoning
capability is due to the lack of 3D spatial knowledge in training data and aim
to solve this problem by training VLMs with Internet-scale spatial reasoning
data. To this end, we present a system to facilitate this approach. We first
develop an automatic 3D spatial VQA data generation framework that scales up to
2 billion VQA examples on 10 million real-world images. We then investigate
various factors in the training recipe, including data quality, training
pipeline, and VLM architecture. Our work features the first internet-scale 3D
spatial reasoning dataset in metric space. By training a VLM on such data, we
significantly enhance its ability on both qualitative and quantitative spatial
VQA. Finally, we demonstrate that this VLM unlocks novel downstream
applications in chain-of-thought spatial reasoning and robotics due to its
quantitative estimation capability. Project website:
https://spatial-vlm.github.io/
A system for training Vision Language Models with internet-scale 3D spatial reasoning data improves their spatial understanding and enables novel applications in quantitative spatial reasoning and robotics.
Vision Language Models VQA 3D spatial reasoning automatic 3D spatial VQA data generation internet-scale 3D spatial reasoning dataset metric space chain-of-thought spatial reasoning