title: MIBench: Evaluating Multimodal Large Language Models over Multiple
    Images
summary: Built on the power of LLMs, numerous multimodal large language models (MLLMs)
  have recently achieved remarkable performance on various vision-language tasks
  across multiple benchmarks. However, most existing MLLMs and benchmarks
  primarily focus on single-image input scenarios, leaving the performance of
  MLLMs when handling realistic multiple images remain underexplored. Although a
  few benchmarks consider multiple images, their evaluation dimensions and
  samples are very limited. Therefore, in this paper, we propose a new benchmark
  MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in
  multi-image scenarios. Specifically, MIBench categorizes the multi-image
  abilities into three scenarios: multi-image instruction (MII), multimodal
  knowledge-seeking (MKS) and multimodal in-context learning (MIC), and
  constructs 13 tasks with a total of 13K annotated samples. During data
  construction, for MII and MKS, we extract correct options from manual
  annotations and create challenging distractors to obtain multiple-choice
  questions. For MIC, to enable an in-depth evaluation, we set four sub-tasks and
  transform the original datasets into in-context learning formats. We evaluate
  several open-source MLLMs and close-source MLLMs on the proposed MIBench. The
  results reveal that although current models excel in single-image tasks, they
  exhibit significant shortcomings when faced with multi-image inputs, such as
  confused fine-grained perception, limited multi-image reasoning, and unstable
  in-context learning. The annotated data in MIBench is available at
  https://huggingface.co/datasets/StarBottle/MIBench.
paper_authors._id: 669f9709cf9bff621ccbb2f0
paper_authors.name: Haowei Liu
paper_authors._id: 669f9709cf9bff621ccbb2f1
paper_authors.name: Xi Zhang
paper_authors._id: 669f9709cf9bff621ccbb2f2
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T11:46:38.953Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 669f9709cf9bff621ccbb2f3
paper_authors.name: Yaya Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:02:48.710Z
paper_authors._id: 63172db0106740aaad4f9954
paper_authors.avatarUrl: /avatars/50f3056c4ba45eed3bbdaced0e1c360f.svg
paper_authors.fullname: Yaya Shi
paper_authors.type: user
paper_authors.user: ShiYaya
paper_authors._id: 669f9709cf9bff621ccbb2f4
paper_authors.name: Chaoya Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:03:02.787Z
paper_authors._id: 61b9976efd429dff1ed3bd44
paper_authors.avatarUrl: /avatars/5cdaa04e970e1e3dcfbb38ba89c0660a.svg
paper_authors.fullname: jiangchaoya
paper_authors.type: user
paper_authors.user: jcy
paper_authors._id: 669f9709cf9bff621ccbb2f5
paper_authors.name: Ming Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T08:11:48.287Z
paper_authors._id: 64771cfdd7cf39f2e9381aa9
paper_authors.avatarUrl: /avatars/48adf00c3b653df02628f80511639e19.svg
paper_authors.fullname: Ming
paper_authors.type: user
paper_authors.user: MingYan123
paper_authors._id: 669f9709cf9bff621ccbb2f6
paper_authors.name: Ji Zhang
paper_authors._id: 669f9709cf9bff621ccbb2f7
paper_authors.name: Fei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-23T12:03:49.360Z
paper_authors._id: 635b8b6a37c6a2c12e2cce00
paper_authors.avatarUrl: /avatars/229fb72180529141515d1df797b33709.svg
paper_authors.fullname: Fei Huang
paper_authors.type: user
paper_authors.user: hzhwcmhf
paper_authors._id: 669f9709cf9bff621ccbb2f8
paper_authors.name: Chunfeng Yuan
paper_authors._id: 669f9709cf9bff621ccbb2f9
paper_authors.name: Bing Li
paper_authors._id: 669f9709cf9bff621ccbb2fa
paper_authors.name: Weiming Hu
paper_title: MIBench: Evaluating Multimodal Large Language Models over Multiple
    Images
paper_summary: Built on the power of LLMs, numerous multimodal large language models (MLLMs)
  have recently achieved remarkable performance on various vision-language tasks
  across multiple benchmarks. However, most existing MLLMs and benchmarks
  primarily focus on single-image input scenarios, leaving the performance of
  MLLMs when handling realistic multiple images remain underexplored. Although a
  few benchmarks consider multiple images, their evaluation dimensions and
  samples are very limited. Therefore, in this paper, we propose a new benchmark
  MIBench, to comprehensively evaluate fine-grained abilities of MLLMs in
  multi-image scenarios. Specifically, MIBench categorizes the multi-image
  abilities into three scenarios: multi-image instruction (MII), multimodal
  knowledge-seeking (MKS) and multimodal in-context learning (MIC), and
  constructs 13 tasks with a total of 13K annotated samples. During data
  construction, for MII and MKS, we extract correct options from manual
  annotations and create challenging distractors to obtain multiple-choice
  questions. For MIC, to enable an in-depth evaluation, we set four sub-tasks and
  transform the original datasets into in-context learning formats. We evaluate
  several open-source MLLMs and close-source MLLMs on the proposed MIBench. The
  results reveal that although current models excel in single-image tasks, they
  exhibit significant shortcomings when faced with multi-image inputs, such as
  confused fine-grained perception, limited multi-image reasoning, and unstable
  in-context learning. The annotated data in MIBench is available at
  https://huggingface.co/datasets/StarBottle/MIBench.
paper_ai_keywords: LLMs (Large Language Models) MLLMs (Multimodal Large Language Models) zero-shot performance vision-language tasks multi-image scenarios MIBench (Multi-Image Benchmark) multi-image instruction (MII) multimodal knowledge-seeking (MKS) multimodal in-context learning (MIC) in-context learning formats multiple-choice questions fine-grained perception multi-image reasoning