title: MuirBench: A Comprehensive Benchmark for Robust Multi-image
    Understanding
summary: We introduce MuirBench, a comprehensive benchmark that focuses on robust
  multi-image understanding capabilities of multimodal LLMs. MuirBench consists
  of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that
  involve 10 categories of multi-image relations (e.g., multiview, temporal
  relations). Comprising 11,264 images and 2,600 multiple-choice questions,
  MuirBench is created in a pairwise manner, where each standard instance is
  paired with an unanswerable variant that has minimal semantic differences, in
  order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our
  results reveal that even the best-performing models like GPT-4o and Gemini Pro
  find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy.
  Open-source multimodal LLMs trained on single images can hardly generalize to
  multi-image questions, hovering below 33.3% in accuracy. These results
  highlight the importance of MuirBench in encouraging the community to develop
  multimodal LLMs that can look beyond a single image, suggesting potential
  pathways for future improvements.
paper_authors._id: 666be4c9f1b3867caf4fef1b
paper_authors.name: Fei Wang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-18T03:32:03.296Z
paper_authors._id: 652d9db6442fb6963b778295
paper_authors.avatarUrl: /avatars/318334e5ddae8fde0cbd06a1b9583ed8.svg
paper_authors.fullname: Fei Wang
paper_authors.type: user
paper_authors.user: fwnlp
paper_authors._id: 666be4c9f1b3867caf4fef1c
paper_authors.name: Xingyu Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:09:39.699Z
paper_authors._id: 6336091b2db86a181ccd6054
paper_authors.avatarUrl: /avatars/829f69436225d05d2c2136bc90f640d7.svg
paper_authors.fullname: Xingyu Fu
paper_authors.type: user
paper_authors.user: Fiaa
paper_authors._id: 666be4c9f1b3867caf4fef1d
paper_authors.name: James Y. Huang
paper_authors._id: 666be4c9f1b3867caf4fef1e
paper_authors.name: Zekun Li
paper_authors._id: 666be4c9f1b3867caf4fef1f
paper_authors.name: Qin Liu
paper_authors._id: 666be4c9f1b3867caf4fef20
paper_authors.name: Xiaogeng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:12:41.886Z
paper_authors._id: 631df96d2899cb403efcf5be
paper_authors.avatarUrl: /avatars/50db06ef7a6efc7b2a525e1c90e576f7.svg
paper_authors.fullname: Xiaogeng Liu
paper_authors.type: user
paper_authors.user: ShletonLiu-N
paper_authors._id: 666be4c9f1b3867caf4fef21
paper_authors.name: Mingyu Derek Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T09:17:23.597Z
paper_authors._id: 644c1df68c51ddbe0ea6ddd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644c1df68c51ddbe0ea6ddd7/KAsOhnOSbrOK54OHtJpiQ.jpeg
paper_authors.fullname: Mingyu Derek Ma
paper_authors.type: user
paper_authors.user: derekma
paper_authors._id: 666be4c9f1b3867caf4fef22
paper_authors.name: Nan Xu
paper_authors._id: 666be4c9f1b3867caf4fef23
paper_authors.name: Wenxuan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:13:15.931Z
paper_authors._id: 619414c7979eb76ec3cc5bab
paper_authors.avatarUrl: /avatars/c1a7055db6150b1bba92e7b13266b489.svg
paper_authors.fullname: Wenxuan Zhou
paper_authors.type: user
paper_authors.user: wzhouad
paper_authors._id: 666be4c9f1b3867caf4fef24
paper_authors.name: Kai Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T07:42:47.727Z
paper_authors._id: 63e0a50242591dda0b9dca5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
paper_authors.fullname: Kai Zhang
paper_authors.type: user
paper_authors.user: drogozhang
paper_authors._id: 666be4c9f1b3867caf4fef25
paper_authors.name: Tianyi Lorena Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:05:46.883Z
paper_authors._id: 63102e82236215d0b7120b0e
paper_authors.avatarUrl: /avatars/232bc42247990c0c4efe5e57176cba85.svg
paper_authors.fullname: Tianyi Lorena Yan
paper_authors.type: user
paper_authors.user: LorenaYannnnn
paper_authors._id: 666be4c9f1b3867caf4fef26
paper_authors.name: Wenjie Jacky Mo
paper_authors._id: 666be4c9f1b3867caf4fef27
paper_authors.name: Hsiang-Hui Liu
paper_authors._id: 666be4c9f1b3867caf4fef28
paper_authors.name: Pan Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:14:31.328Z
paper_authors._id: 60f5f68fa7fd83d025749234
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
paper_authors.fullname: Pan Lu
paper_authors.type: user
paper_authors.user: lupantech
paper_authors._id: 666be4c9f1b3867caf4fef29
paper_authors.name: Chunyuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:14:24.265Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 666be4c9f1b3867caf4fef2a
paper_authors.name: Chaowei Xiao
paper_authors._id: 666be4c9f1b3867caf4fef2b
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:14:10.075Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 666be4c9f1b3867caf4fef2c
paper_authors.name: Dan Roth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:13:50.925Z
paper_authors._id: 63191aa1f18d0b5d999a47fc
paper_authors.avatarUrl: /avatars/d7eadb22aa3d8a067189b924767864e9.svg
paper_authors.fullname: Dan Roth
paper_authors.type: user
paper_authors.user: danielpaulroth
paper_authors._id: 666be4c9f1b3867caf4fef2d
paper_authors.name: Sheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:33:51.021Z
paper_authors._id: 6234c11b7d5de9839bc44163
paper_authors.avatarUrl: /avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
paper_authors.fullname: Sheng Zhang
paper_authors.type: user
paper_authors.user: shengz
paper_authors._id: 666be4c9f1b3867caf4fef2e
paper_authors.name: Hoifung Poon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T10:13:43.542Z
paper_authors._id: 664d07456083f276c4feb1a4
paper_authors.avatarUrl: /avatars/1bfa6d8f82e9223b47630cefd79d7d0e.svg
paper_authors.fullname: Hoifung Poon
paper_authors.type: user
paper_authors.user: hoifung
paper_authors._id: 666be4c9f1b3867caf4fef2f
paper_authors.name: Muhao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:21:36.017Z
paper_authors._id: 666be8ef81f01fbd60e84f01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/666be8ef81f01fbd60e84f01/PiCUqr7XT96HGpa_GVRLr.jpeg
paper_authors.fullname: Muhao Chen
paper_authors.type: user
paper_authors.user: Muhao
paper_title: MuirBench: A Comprehensive Benchmark for Robust Multi-image
    Understanding
paper_summary: We introduce MuirBench, a comprehensive benchmark that focuses on robust
  multi-image understanding capabilities of multimodal LLMs. MuirBench consists
  of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that
  involve 10 categories of multi-image relations (e.g., multiview, temporal
  relations). Comprising 11,264 images and 2,600 multiple-choice questions,
  MuirBench is created in a pairwise manner, where each standard instance is
  paired with an unanswerable variant that has minimal semantic differences, in
  order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our
  results reveal that even the best-performing models like GPT-4o and Gemini Pro
  find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy.
  Open-source multimodal LLMs trained on single images can hardly generalize to
  multi-image questions, hovering below 33.3% in accuracy. These results
  highlight the importance of MuirBench in encouraging the community to develop
  multimodal LLMs that can look beyond a single image, suggesting potential
  pathways for future improvements.
paper_ai_summary: MuirBench is a benchmark for evaluating robust multi-image understanding in multimodal LLMs, highlighting challenges and the need for improvements in handling multi-image tasks.
paper_ai_keywords: multimodal LLMs scene understanding ordering multiview relations temporal relations pairwise manner unanswerable variant GPT-4o Gemini Pro