title: MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence
summary: Spatial intelligence is essential for multimodal large language models
  (MLLMs) operating in the complex physical world. Existing benchmarks, however,
  probe only single-image relations and thus fail to assess the multi-image
  spatial reasoning that real-world deployments demand. We introduce MMSI-Bench,
  a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision
  researchers spent more than 300 hours meticulously crafting 1,000 challenging,
  unambiguous multiple-choice questions from over 120,000 images, each paired
  with carefully designed distractors and a step-by-step reasoning process. We
  conduct extensive experiments and thoroughly evaluate 34 open-source and
  proprietary MLLMs, observing a wide gap: the strongest open-source model
  attains roughly 30% accuracy and OpenAI's o3 reasoning model reaches 40%, while
  humans score 97%. These results underscore the challenging nature of MMSI-Bench
  and the substantial headroom for future research. Leveraging the annotated
  reasoning processes, we also provide an automated error analysis pipeline that
  diagnoses four dominant failure modes, including (1) grounding errors, (2)
  overlap-matching and scene-reconstruction errors, (3) situation-transformation
  reasoning errors, and (4) spatial-logic errors, offering valuable insights for
  advancing multi-image spatial intelligence. Project page:
  https://runsenxu.com/projects/MMSI_Bench .
paper_authors._id: 6839f2b800f90dff0cff5f6c
paper_authors.name: Sihan Yang
paper_authors._id: 6839f2b800f90dff0cff5f6d
paper_authors.name: Runsen Xu
paper_authors._id: 6839f2b800f90dff0cff5f6e
paper_authors.name: Yiman Xie
paper_authors._id: 6839f2b800f90dff0cff5f6f
paper_authors.name: Sizhe Yang
paper_authors._id: 6839f2b800f90dff0cff5f70
paper_authors.name: Mo Li
paper_authors._id: 6839f2b800f90dff0cff5f71
paper_authors.name: Jingli Lin
paper_authors._id: 6839f2b800f90dff0cff5f72
paper_authors.name: Chenming Zhu
paper_authors._id: 6839f2b800f90dff0cff5f73
paper_authors.name: Xiaochen Chen
paper_authors._id: 6839f2b800f90dff0cff5f74
paper_authors.name: Haodong Duan
paper_authors._id: 6839f2b800f90dff0cff5f75
paper_authors.name: Xiangyu Yue
paper_authors._id: 6839f2b800f90dff0cff5f76
paper_authors.name: Dahua Lin
paper_authors._id: 6839f2b800f90dff0cff5f77
paper_authors.name: Tai Wang
paper_authors._id: 6839f2b800f90dff0cff5f78
paper_authors.name: Jiangmiao Pang
paper_title: MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence
paper_summary: Spatial intelligence is essential for multimodal large language models
  (MLLMs) operating in the complex physical world. Existing benchmarks, however,
  probe only single-image relations and thus fail to assess the multi-image
  spatial reasoning that real-world deployments demand. We introduce MMSI-Bench,
  a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision
  researchers spent more than 300 hours meticulously crafting 1,000 challenging,
  unambiguous multiple-choice questions from over 120,000 images, each paired
  with carefully designed distractors and a step-by-step reasoning process. We
  conduct extensive experiments and thoroughly evaluate 34 open-source and
  proprietary MLLMs, observing a wide gap: the strongest open-source model
  attains roughly 30% accuracy and OpenAI's o3 reasoning model reaches 40%, while
  humans score 97%. These results underscore the challenging nature of MMSI-Bench
  and the substantial headroom for future research. Leveraging the annotated
  reasoning processes, we also provide an automated error analysis pipeline that
  diagnoses four dominant failure modes, including (1) grounding errors, (2)
  overlap-matching and scene-reconstruction errors, (3) situation-transformation
  reasoning errors, and (4) spatial-logic errors, offering valuable insights for
  advancing multi-image spatial intelligence. Project page:
  https://runsenxu.com/projects/MMSI_Bench .
paper_ai_summary: MMSI-Bench, a VQA benchmark for multi-image spatial intelligence, reveals significant gaps in multimodal large language models' performance compared to human accuracy.
paper_ai_keywords: VQA MMSI-Bench 3D-vision multimodal large language models multiple-choice questions distractors reasoning processes grounding errors overlap-matching scene-reconstruction errors situation-transformation reasoning errors spatial-logic errors