MMIU: Multimodal Multi-image Understanding for Evaluating Large
  Vision-Language Models
The capability to process multiple images is crucial for Large
Vision-Language Models (LVLMs) to develop a more thorough and nuanced
understanding of a scene. Recent multi-image LVLMs have begun to address this
need. However, their evaluation has not kept pace with their development. To
fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU)
benchmark, a comprehensive evaluation suite designed to assess LVLMs across a
wide range of multi-image tasks. MMIU encompasses 7 types of multi-image
relationships, 52 tasks, 77K images, and 11K meticulously curated
multiple-choice questions, making it the most extensive benchmark of its kind.
Our evaluation of 24 popular LVLMs, including both open-source and proprietary
models, reveals significant challenges in multi-image comprehension,
particularly in tasks involving spatial understanding. Even the most advanced
models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through
multi-faceted analytical experiments, we identify key performance gaps and
limitations, providing valuable insights for future model and data
improvements. We aim for MMIU to advance the frontier of LVLM research and
development, moving us toward achieving sophisticated multimodal multi-image
user interactions.
66b2d0f6213dafdcb8e2d2bb
Fanqing Meng
admin_assigned
2024-08-08T10:57:03.952Z
640b37b2bab5ca8fbe7df8f2
/avatars/c7bef45efad6a0d911a720e2236fcba5.svg
fanqing meng
user
FanqingM
66b2d0f6213dafdcb8e2d2bc
Jin Wang
66b2d0f6213dafdcb8e2d2bd
Chuanhao Li
claimed_verified
2025-06-23T08:34:53.442Z
674ea59a8f2e7614a6c72f26
/avatars/86fd4c6d7d33435de49b35659bf65265.svg
Chuanhao
user
ChuanhaoLi
66b2d0f6213dafdcb8e2d2be
Quanfeng Lu
admin_assigned
2024-08-08T10:57:16.135Z
653a483dacdeea08424ef55d
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tYG2isIZBLCaBbWABatgK.png
Quanfeng Lu
user
hflqf88888
66b2d0f6213dafdcb8e2d2bf
Hao Tian
66b2d0f6213dafdcb8e2d2c0
Jiaqi Liao
admin_assigned
2024-08-08T10:57:30.540Z
6630b287ce65a66ed8f04eba
/avatars/18ad0e3b453021ea6006e23333e92c12.svg
liaojiaqi
user
ljq940913
66b2d0f6213dafdcb8e2d2c1
Xizhou Zhu
admin_assigned
2024-08-08T10:57:36.015Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
66b2d0f6213dafdcb8e2d2c2
Jifeng Dai
admin_assigned
2024-08-08T10:57:42.021Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
66b2d0f6213dafdcb8e2d2c3
Yu Qiao
66b2d0f6213dafdcb8e2d2c4
Ping Luo
66b2d0f6213dafdcb8e2d2c5
Kaipeng Zhang
admin_assigned
2024-08-08T10:57:53.508Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
66b2d0f6213dafdcb8e2d2c6
Wenqi Shao
admin_assigned
2024-08-08T10:57:47.839Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
MMIU: Multimodal Multi-image Understanding for Evaluating Large
  Vision-Language Models
The capability to process multiple images is crucial for Large
Vision-Language Models (LVLMs) to develop a more thorough and nuanced
understanding of a scene. Recent multi-image LVLMs have begun to address this
need. However, their evaluation has not kept pace with their development. To
fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU)
benchmark, a comprehensive evaluation suite designed to assess LVLMs across a
wide range of multi-image tasks. MMIU encompasses 7 types of multi-image
relationships, 52 tasks, 77K images, and 11K meticulously curated
multiple-choice questions, making it the most extensive benchmark of its kind.
Our evaluation of 24 popular LVLMs, including both open-source and proprietary
models, reveals significant challenges in multi-image comprehension,
particularly in tasks involving spatial understanding. Even the most advanced
models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through
multi-faceted analytical experiments, we identify key performance gaps and
limitations, providing valuable insights for future model and data
improvements. We aim for MMIU to advance the frontier of LVLM research and
development, moving us toward achieving sophisticated multimodal multi-image
user interactions.
The Multimodal Multi-image Understanding (MMIU) benchmark assesses LVLMs across various multi-image tasks, highlighting challenges in spatial understanding.
Large Vision-Language Models (LVLMs) Multimodal Multi-image Understanding (MMIU) benchmark multi-image tasks multiple-choice questions spatial understanding