title: MMIU: Multimodal Multi-image Understanding for Evaluating Large
    Vision-Language Models
summary: The capability to process multiple images is crucial for Large
  Vision-Language Models (LVLMs) to develop a more thorough and nuanced
  understanding of a scene. Recent multi-image LVLMs have begun to address this
  need. However, their evaluation has not kept pace with their development. To
  fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU)
  benchmark, a comprehensive evaluation suite designed to assess LVLMs across a
  wide range of multi-image tasks. MMIU encompasses 7 types of multi-image
  relationships, 52 tasks, 77K images, and 11K meticulously curated
  multiple-choice questions, making it the most extensive benchmark of its kind.
  Our evaluation of 24 popular LVLMs, including both open-source and proprietary
  models, reveals significant challenges in multi-image comprehension,
  particularly in tasks involving spatial understanding. Even the most advanced
  models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through
  multi-faceted analytical experiments, we identify key performance gaps and
  limitations, providing valuable insights for future model and data
  improvements. We aim for MMIU to advance the frontier of LVLM research and
  development, moving us toward achieving sophisticated multimodal multi-image
  user interactions.
paper_authors._id: 66b2d0f6213dafdcb8e2d2bb
paper_authors.name: Fanqing Meng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:03.952Z
paper_authors._id: 640b37b2bab5ca8fbe7df8f2
paper_authors.avatarUrl: /avatars/c7bef45efad6a0d911a720e2236fcba5.svg
paper_authors.fullname: fanqing meng
paper_authors.type: user
paper_authors.user: FanqingM
paper_authors._id: 66b2d0f6213dafdcb8e2d2bc
paper_authors.name: Jin Wang
paper_authors._id: 66b2d0f6213dafdcb8e2d2bd
paper_authors.name: Chuanhao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:11.008Z
paper_authors._id: 6533f7ecb3852ed1ceb48e47
paper_authors.avatarUrl: /avatars/5d767c093e73f06a89f625c3a5903902.svg
paper_authors.fullname: Chuanhao Li
paper_authors.type: user
paper_authors.user: cyrilli
paper_authors._id: 66b2d0f6213dafdcb8e2d2be
paper_authors.name: Quanfeng Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:16.135Z
paper_authors._id: 653a483dacdeea08424ef55d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tYG2isIZBLCaBbWABatgK.png
paper_authors.fullname: Quanfeng Lu
paper_authors.type: user
paper_authors.user: hflqf88888
paper_authors._id: 66b2d0f6213dafdcb8e2d2bf
paper_authors.name: Hao Tian
paper_authors._id: 66b2d0f6213dafdcb8e2d2c0
paper_authors.name: Jiaqi Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:30.540Z
paper_authors._id: 6630b287ce65a66ed8f04eba
paper_authors.avatarUrl: /avatars/18ad0e3b453021ea6006e23333e92c12.svg
paper_authors.fullname: liaojiaqi
paper_authors.type: user
paper_authors.user: ljq940913
paper_authors._id: 66b2d0f6213dafdcb8e2d2c1
paper_authors.name: Xizhou Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:36.015Z
paper_authors._id: 64ae2359179421d320b1694b
paper_authors.avatarUrl: /avatars/c387a75191005bcaa473091de5383a10.svg
paper_authors.fullname: Xizhou Zhu
paper_authors.type: user
paper_authors.user: Einsiedler
paper_authors._id: 66b2d0f6213dafdcb8e2d2c2
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:42.021Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_authors._id: 66b2d0f6213dafdcb8e2d2c3
paper_authors.name: Yu Qiao
paper_authors._id: 66b2d0f6213dafdcb8e2d2c4
paper_authors.name: Ping Luo
paper_authors._id: 66b2d0f6213dafdcb8e2d2c5
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:53.508Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_authors._id: 66b2d0f6213dafdcb8e2d2c6
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:57:47.839Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_title: MMIU: Multimodal Multi-image Understanding for Evaluating Large
    Vision-Language Models
paper_summary: The capability to process multiple images is crucial for Large
  Vision-Language Models (LVLMs) to develop a more thorough and nuanced
  understanding of a scene. Recent multi-image LVLMs have begun to address this
  need. However, their evaluation has not kept pace with their development. To
  fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU)
  benchmark, a comprehensive evaluation suite designed to assess LVLMs across a
  wide range of multi-image tasks. MMIU encompasses 7 types of multi-image
  relationships, 52 tasks, 77K images, and 11K meticulously curated
  multiple-choice questions, making it the most extensive benchmark of its kind.
  Our evaluation of 24 popular LVLMs, including both open-source and proprietary
  models, reveals significant challenges in multi-image comprehension,
  particularly in tasks involving spatial understanding. Even the most advanced
  models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through
  multi-faceted analytical experiments, we identify key performance gaps and
  limitations, providing valuable insights for future model and data
  improvements. We aim for MMIU to advance the frontier of LVLM research and
  development, moving us toward achieving sophisticated multimodal multi-image
  user interactions.
paper_ai_keywords: Large Vision-Language Models (LVLMs) multi-image LVLMs Multimodal Multi-image Understanding (MMIU) multi-image relationships multiple-choice questions GPT-4o