MuirBench: A Comprehensive Benchmark for Robust Multi-image
  Understanding
We introduce MuirBench, a comprehensive benchmark that focuses on robust
multi-image understanding capabilities of multimodal LLMs. MuirBench consists
of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that
involve 10 categories of multi-image relations (e.g., multiview, temporal
relations). Comprising 11,264 images and 2,600 multiple-choice questions,
MuirBench is created in a pairwise manner, where each standard instance is
paired with an unanswerable variant that has minimal semantic differences, in
order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our
results reveal that even the best-performing models like GPT-4o and Gemini Pro
find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy.
Open-source multimodal LLMs trained on single images can hardly generalize to
multi-image questions, hovering below 33.3% in accuracy. These results
highlight the importance of MuirBench in encouraging the community to develop
multimodal LLMs that can look beyond a single image, suggesting potential
pathways for future improvements.
666be4c9f1b3867caf4fef1b
Fei Wang
extracted_confirmed
2024-06-18T03:32:03.296Z
652d9db6442fb6963b778295
/avatars/318334e5ddae8fde0cbd06a1b9583ed8.svg
Fei Wang
user
fwnlp
666be4c9f1b3867caf4fef1c
Xingyu Fu
admin_assigned
2024-06-14T10:09:39.699Z
6336091b2db86a181ccd6054
/avatars/829f69436225d05d2c2136bc90f640d7.svg
Xingyu Fu
user
Fiaa
666be4c9f1b3867caf4fef1d
James Y. Huang
666be4c9f1b3867caf4fef1e
Zekun Li
666be4c9f1b3867caf4fef1f
Qin Liu
666be4c9f1b3867caf4fef20
Xiaogeng Liu
admin_assigned
2024-06-14T10:12:41.886Z
631df96d2899cb403efcf5be
/avatars/50db06ef7a6efc7b2a525e1c90e576f7.svg
Xiaogeng Liu
user
ShletonLiu-N
666be4c9f1b3867caf4fef21
Mingyu Derek Ma
claimed_verified
2024-06-14T09:17:23.597Z
644c1df68c51ddbe0ea6ddd7
https://cdn-avatars.huggingface.co/v1/production/uploads/644c1df68c51ddbe0ea6ddd7/KAsOhnOSbrOK54OHtJpiQ.jpeg
Mingyu Derek Ma
user
derekma
666be4c9f1b3867caf4fef22
Nan Xu
666be4c9f1b3867caf4fef23
Wenxuan Zhou
admin_assigned
2024-06-14T10:13:15.931Z
619414c7979eb76ec3cc5bab
/avatars/c1a7055db6150b1bba92e7b13266b489.svg
Wenxuan Zhou
user
wzhouad
666be4c9f1b3867caf4fef24
Kai Zhang
claimed_verified
2024-08-08T07:42:47.727Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
user
drogozhang
666be4c9f1b3867caf4fef25
Tianyi Lorena Yan
claimed_verified
2025-03-10T08:05:46.883Z
63102e82236215d0b7120b0e
/avatars/232bc42247990c0c4efe5e57176cba85.svg
Tianyi Lorena Yan
user
LorenaYannnnn
666be4c9f1b3867caf4fef26
Wenjie Jacky Mo
666be4c9f1b3867caf4fef27
Hsiang-Hui Liu
666be4c9f1b3867caf4fef28
Pan Lu
admin_assigned
2024-06-14T10:14:31.328Z
60f5f68fa7fd83d025749234
https://cdn-avatars.huggingface.co/v1/production/uploads/60f5f68fa7fd83d025749234/gCeJAZfzaANAcEvI6v5-P.jpeg
Pan Lu
user
lupantech
666be4c9f1b3867caf4fef29
Chunyuan Li
admin_assigned
2024-06-14T10:14:24.265Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
666be4c9f1b3867caf4fef2a
Chaowei Xiao
666be4c9f1b3867caf4fef2b
Kai-Wei Chang
admin_assigned
2024-06-14T10:14:10.075Z
60b7b9d71b90c5d07c23fbd0
https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
Kai-Wei Chang
user
kaiweichang
666be4c9f1b3867caf4fef2c
Dan Roth
admin_assigned
2024-06-14T10:13:50.925Z
63191aa1f18d0b5d999a47fc
/avatars/d7eadb22aa3d8a067189b924767864e9.svg
Dan Roth
user
danielpaulroth
666be4c9f1b3867caf4fef2d
Sheng Zhang
claimed_verified
2024-06-17T07:33:51.021Z
6234c11b7d5de9839bc44163
/avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
Sheng Zhang
user
shengz
666be4c9f1b3867caf4fef2e
Hoifung Poon
admin_assigned
2024-06-14T10:13:43.542Z
664d07456083f276c4feb1a4
/avatars/1bfa6d8f82e9223b47630cefd79d7d0e.svg
Hoifung Poon
user
hoifung
666be4c9f1b3867caf4fef2f
Muhao Chen
claimed_verified
2024-06-14T07:21:36.017Z
666be8ef81f01fbd60e84f01
https://cdn-avatars.huggingface.co/v1/production/uploads/666be8ef81f01fbd60e84f01/PiCUqr7XT96HGpa_GVRLr.jpeg
Muhao Chen
user
Muhao
MuirBench: A Comprehensive Benchmark for Robust Multi-image
  Understanding
We introduce MuirBench, a comprehensive benchmark that focuses on robust
multi-image understanding capabilities of multimodal LLMs. MuirBench consists
of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that
involve 10 categories of multi-image relations (e.g., multiview, temporal
relations). Comprising 11,264 images and 2,600 multiple-choice questions,
MuirBench is created in a pairwise manner, where each standard instance is
paired with an unanswerable variant that has minimal semantic differences, in
order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our
results reveal that even the best-performing models like GPT-4o and Gemini Pro
find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy.
Open-source multimodal LLMs trained on single images can hardly generalize to
multi-image questions, hovering below 33.3% in accuracy. These results
highlight the importance of MuirBench in encouraging the community to develop
multimodal LLMs that can look beyond a single image, suggesting potential
pathways for future improvements.
MuirBench is a benchmark for evaluating robust multi-image understanding in multimodal LLMs, highlighting challenges and the need for improvements in handling multi-image tasks.
multimodal LLMs scene understanding ordering multiview relations temporal relations pairwise manner unanswerable variant GPT-4o Gemini Pro