MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence
Spatial intelligence is essential for multimodal large language models
(MLLMs) operating in the complex physical world. Existing benchmarks, however,
probe only single-image relations and thus fail to assess the multi-image
spatial reasoning that real-world deployments demand. We introduce MMSI-Bench,
a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision
researchers spent more than 300 hours meticulously crafting 1,000 challenging,
unambiguous multiple-choice questions from over 120,000 images, each paired
with carefully designed distractors and a step-by-step reasoning process. We
conduct extensive experiments and thoroughly evaluate 34 open-source and
proprietary MLLMs, observing a wide gap: the strongest open-source model
attains roughly 30% accuracy and OpenAI's o3 reasoning model reaches 40%, while
humans score 97%. These results underscore the challenging nature of MMSI-Bench
and the substantial headroom for future research. Leveraging the annotated
reasoning processes, we also provide an automated error analysis pipeline that
diagnoses four dominant failure modes, including (1) grounding errors, (2)
overlap-matching and scene-reconstruction errors, (3) situation-transformation
reasoning errors, and (4) spatial-logic errors, offering valuable insights for
advancing multi-image spatial intelligence. Project page:
https://runsenxu.com/projects/MMSI_Bench .
6839f2b800f90dff0cff5f6c
Sihan Yang
6839f2b800f90dff0cff5f6d
Runsen Xu
6839f2b800f90dff0cff5f6e
Yiman Xie
6839f2b800f90dff0cff5f6f
Sizhe Yang
6839f2b800f90dff0cff5f70
Mo Li
6839f2b800f90dff0cff5f71
Jingli Lin
6839f2b800f90dff0cff5f72
Chenming Zhu
6839f2b800f90dff0cff5f73
Xiaochen Chen
6839f2b800f90dff0cff5f74
Haodong Duan
6839f2b800f90dff0cff5f75
Xiangyu Yue
6839f2b800f90dff0cff5f76
Dahua Lin
6839f2b800f90dff0cff5f77
Tai Wang
6839f2b800f90dff0cff5f78
Jiangmiao Pang
MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence
Spatial intelligence is essential for multimodal large language models
(MLLMs) operating in the complex physical world. Existing benchmarks, however,
probe only single-image relations and thus fail to assess the multi-image
spatial reasoning that real-world deployments demand. We introduce MMSI-Bench,
a VQA benchmark dedicated to multi-image spatial intelligence. Six 3D-vision
researchers spent more than 300 hours meticulously crafting 1,000 challenging,
unambiguous multiple-choice questions from over 120,000 images, each paired
with carefully designed distractors and a step-by-step reasoning process. We
conduct extensive experiments and thoroughly evaluate 34 open-source and
proprietary MLLMs, observing a wide gap: the strongest open-source model
attains roughly 30% accuracy and OpenAI's o3 reasoning model reaches 40%, while
humans score 97%. These results underscore the challenging nature of MMSI-Bench
and the substantial headroom for future research. Leveraging the annotated
reasoning processes, we also provide an automated error analysis pipeline that
diagnoses four dominant failure modes, including (1) grounding errors, (2)
overlap-matching and scene-reconstruction errors, (3) situation-transformation
reasoning errors, and (4) spatial-logic errors, offering valuable insights for
advancing multi-image spatial intelligence. Project page:
https://runsenxu.com/projects/MMSI_Bench .
MMSI-Bench, a VQA benchmark for multi-image spatial intelligence, reveals significant gaps in multimodal large language models' performance compared to human accuracy.
VQA MMSI-Bench 3D-vision multimodal large language models multiple-choice questions distractors reasoning processes grounding errors overlap-matching scene-reconstruction errors situation-transformation reasoning errors spatial-logic errors