Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal
  Large Language Models
Multi-modal large language models (MLLMs) have rapidly advanced in visual
tasks, yet their spatial understanding remains limited to single images,
leaving them ill-suited for robotics and other real-world applications that
require multi-frame reasoning. In this paper, we propose a framework to equip
MLLMs with robust multi-frame spatial understanding by integrating depth
perception, visual correspondence, and dynamic perception. Central to our
approach is the MultiSPA dataset, a novel, large-scale collection of more than
27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we
introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks
under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves
significant gains over baselines and proprietary systems, demonstrating
scalable, generalizable multi-frame reasoning. We further observe multi-task
benefits and early indications of emergent capabilities in challenging
scenarios, and showcase how our model can serve as a multi-frame reward
annotator for robotics.
683052ef14801638dfa857bc
Runsen Xu
683052ef14801638dfa857bd
Weiyao Wang
683052ef14801638dfa857be
Hao Tang
683052ef14801638dfa857bf
Xingyu Chen
683052ef14801638dfa857c0
Xiaodong Wang
683052ef14801638dfa857c1
Fu-Jen Chu
683052ef14801638dfa857c2
Dahua Lin
683052ef14801638dfa857c3
Matt Feiszli
683052ef14801638dfa857c4
Kevin J. Liang
Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal
  Large Language Models
Multi-modal large language models (MLLMs) have rapidly advanced in visual
tasks, yet their spatial understanding remains limited to single images,
leaving them ill-suited for robotics and other real-world applications that
require multi-frame reasoning. In this paper, we propose a framework to equip
MLLMs with robust multi-frame spatial understanding by integrating depth
perception, visual correspondence, and dynamic perception. Central to our
approach is the MultiSPA dataset, a novel, large-scale collection of more than
27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we
introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks
under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves
significant gains over baselines and proprietary systems, demonstrating
scalable, generalizable multi-frame reasoning. We further observe multi-task
benefits and early indications of emergent capabilities in challenging
scenarios, and showcase how our model can serve as a multi-frame reward
annotator for robotics.
Multi-SpatialMLLM framework enhances MLLMs with multi-frame spatial understanding through depth perception, visual correspondence, and dynamic perception, achieving significant gains in multi-frame reasoning tasks.
multi-modal large language models depth perception visual correspondence dynamic perception MultiSPA dataset multi-frame spatial understanding multi-frame reasoning multi-task benefits reward annotator