title: Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal
    Large Language Models
summary: Multi-modal large language models (MLLMs) have rapidly advanced in visual
  tasks, yet their spatial understanding remains limited to single images,
  leaving them ill-suited for robotics and other real-world applications that
  require multi-frame reasoning. In this paper, we propose a framework to equip
  MLLMs with robust multi-frame spatial understanding by integrating depth
  perception, visual correspondence, and dynamic perception. Central to our
  approach is the MultiSPA dataset, a novel, large-scale collection of more than
  27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we
  introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks
  under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves
  significant gains over baselines and proprietary systems, demonstrating
  scalable, generalizable multi-frame reasoning. We further observe multi-task
  benefits and early indications of emergent capabilities in challenging
  scenarios, and showcase how our model can serve as a multi-frame reward
  annotator for robotics.
paper_authors._id: 683052ef14801638dfa857bc
paper_authors.name: Runsen Xu
paper_authors._id: 683052ef14801638dfa857bd
paper_authors.name: Weiyao Wang
paper_authors._id: 683052ef14801638dfa857be
paper_authors.name: Hao Tang
paper_authors._id: 683052ef14801638dfa857bf
paper_authors.name: Xingyu Chen
paper_authors._id: 683052ef14801638dfa857c0
paper_authors.name: Xiaodong Wang
paper_authors._id: 683052ef14801638dfa857c1
paper_authors.name: Fu-Jen Chu
paper_authors._id: 683052ef14801638dfa857c2
paper_authors.name: Dahua Lin
paper_authors._id: 683052ef14801638dfa857c3
paper_authors.name: Matt Feiszli
paper_authors._id: 683052ef14801638dfa857c4
paper_authors.name: Kevin J. Liang
paper_title: Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal
    Large Language Models
paper_summary: Multi-modal large language models (MLLMs) have rapidly advanced in visual
  tasks, yet their spatial understanding remains limited to single images,
  leaving them ill-suited for robotics and other real-world applications that
  require multi-frame reasoning. In this paper, we propose a framework to equip
  MLLMs with robust multi-frame spatial understanding by integrating depth
  perception, visual correspondence, and dynamic perception. Central to our
  approach is the MultiSPA dataset, a novel, large-scale collection of more than
  27 million samples spanning diverse 3D and 4D scenes. Alongside MultiSPA, we
  introduce a comprehensive benchmark that tests a wide spectrum of spatial tasks
  under uniform metrics. Our resulting model, Multi-SpatialMLLM, achieves
  significant gains over baselines and proprietary systems, demonstrating
  scalable, generalizable multi-frame reasoning. We further observe multi-task
  benefits and early indications of emergent capabilities in challenging
  scenarios, and showcase how our model can serve as a multi-frame reward
  annotator for robotics.
paper_ai_summary: Multi-SpatialMLLM framework enhances MLLMs with multi-frame spatial understanding through depth perception, visual correspondence, and dynamic perception, achieving significant gains in multi-frame reasoning tasks.
paper_ai_keywords: multi-modal large language models depth perception visual correspondence dynamic perception MultiSPA dataset multi-frame spatial understanding multi-frame reasoning multi-task benefits reward annotator