title: MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large
    Vision-Language Models
summary: Interleaved multimodal comprehension and generation, enabling models to
  produce and interpret both images and text in arbitrary sequences, have become
  a pivotal area in multimodal learning. Despite significant advancements, the
  evaluation of this capability remains insufficient. Existing benchmarks suffer
  from limitations in data scale, scope, and evaluation depth, while current
  evaluation metrics are often costly or biased, lacking in reliability for
  practical applications. To address these challenges, we introduce MMIE, a
  large-scale knowledge-intensive benchmark for evaluating interleaved multimodal
  comprehension and generation in Large Vision-Language Models (LVLMs). MMIE
  comprises 20K meticulously curated multimodal queries, spanning 3 categories,
  12 fields, and 102 subfields, including mathematics, coding, physics,
  literature, health, and arts. It supports both interleaved inputs and outputs,
  offering a mix of multiple-choice and open-ended question formats to evaluate
  diverse competencies. Moreover, we propose a reliable automated evaluation
  metric, leveraging a scoring model fine-tuned with human-annotated data and
  systematic evaluation criteria, aimed at reducing bias and improving evaluation
  accuracy. Extensive experiments demonstrate the effectiveness of our benchmark
  and metrics in providing a comprehensive evaluation of interleaved LVLMs.
  Specifically, we evaluate eight LVLMs, revealing that even the best models show
  significant room for improvement, with most achieving only moderate results. We
  believe MMIE will drive further advancements in the development of interleaved
  LVLMs. We publicly release our benchmark and code in
  https://mmie-bench.github.io/.
paper_authors._id: 670dc9101fd338212b902556
paper_authors.name: Peng Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:03.755Z
paper_authors._id: 643e9ee6f6bb3c31a26e7bc4
paper_authors.avatarUrl: /avatars/acfaa7d6a23dada24c86b954c3be116a.svg
paper_authors.fullname: Peng Xia
paper_authors.type: user
paper_authors.user: richardxp888
paper_authors._id: 670dc9101fd338212b902557
paper_authors.name: Siwei Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T08:37:39.675Z
paper_authors._id: 65941852f0152a21fc860f79
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65941852f0152a21fc860f79/mHOkrMOkivxxK__F4AK2o.png
paper_authors.fullname: Lillianwei
paper_authors.type: user
paper_authors.user: Lillianwei
paper_authors._id: 670dc9101fd338212b902558
paper_authors.name: Shi Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:02.150Z
paper_authors._id: 6644bb2c9bdbd85493074411
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6644bb2c9bdbd85493074411/iug8dkP1zjID-kXzNxDjD.jpeg
paper_authors.fullname: SHI QIU
paper_authors.type: user
paper_authors.user: StarThomas1002
paper_authors._id: 670dc9101fd338212b902559
paper_authors.name: Yiyang Zhou
paper_authors._id: 670dc9101fd338212b90255a
paper_authors.name: Zhaoyang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:26:59.440Z
paper_authors._id: 66c719e2f3f9994f2ae6c1d2
paper_authors.avatarUrl: /avatars/c6cbbf458e30d235bc3db12628510fc4.svg
paper_authors.fullname: zhaoyang.wang
paper_authors.type: user
paper_authors.user: wangzhaoyang
paper_authors._id: 670dc9101fd338212b90255b
paper_authors.name: Wenhao Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:27:06.091Z
paper_authors._id: 65d0c9b4fff50114959c0004
paper_authors.avatarUrl: /avatars/31f599204e51528ec317aeb980f4676e.svg
paper_authors.fullname: Wenhao Zheng
paper_authors.type: user
paper_authors.user: WendellZwh
paper_authors._id: 670dc9101fd338212b90255c
paper_authors.name: Zhaorun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:27:12.112Z
paper_authors._id: 64d448d3d095077728f7c740
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d448d3d095077728f7c740/4CFkiVoXXDsu7uQPTU0n3.jpeg
paper_authors.fullname: Zhaorun Chen
paper_authors.type: user
paper_authors.user: Zhaorun
paper_authors._id: 670dc9101fd338212b90255d
paper_authors.name: Chenhang Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:27:18.302Z
paper_authors._id: 64a26627f09ad861c85487e0
paper_authors.avatarUrl: /avatars/baefd153c59f3a3238fd6034c6daf0d4.svg
paper_authors.fullname: ChenhangCui
paper_authors.type: user
paper_authors.user: Chenhangcui
paper_authors._id: 670dc9101fd338212b90255e
paper_authors.name: Mingyu Ding
paper_authors._id: 670dc9101fd338212b90255f
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:27:58.086Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 670dc9101fd338212b902560
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:28:14.317Z
paper_authors._id: 6672e20d1dbdf7da8310dd92
paper_authors.avatarUrl: /avatars/5d2fb23f92a7f9ff025a5be17a26de4d.svg
paper_authors.fullname: lijuanwang
paper_authors.type: user
paper_authors.user: lijuanwang228
paper_authors._id: 670dc9101fd338212b902561
paper_authors.name: Huaxiu Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:28:27.413Z
paper_authors._id: 65f667a455009c4ad9e6ac4c
paper_authors.avatarUrl: /avatars/ed10c2cf2ba3fde6d7da93f076961607.svg
paper_authors.fullname: Yao
paper_authors.type: user
paper_authors.user: Huaxiu
paper_title: MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large
    Vision-Language Models
paper_summary: Interleaved multimodal comprehension and generation, enabling models to
  produce and interpret both images and text in arbitrary sequences, have become
  a pivotal area in multimodal learning. Despite significant advancements, the
  evaluation of this capability remains insufficient. Existing benchmarks suffer
  from limitations in data scale, scope, and evaluation depth, while current
  evaluation metrics are often costly or biased, lacking in reliability for
  practical applications. To address these challenges, we introduce MMIE, a
  large-scale knowledge-intensive benchmark for evaluating interleaved multimodal
  comprehension and generation in Large Vision-Language Models (LVLMs). MMIE
  comprises 20K meticulously curated multimodal queries, spanning 3 categories,
  12 fields, and 102 subfields, including mathematics, coding, physics,
  literature, health, and arts. It supports both interleaved inputs and outputs,
  offering a mix of multiple-choice and open-ended question formats to evaluate
  diverse competencies. Moreover, we propose a reliable automated evaluation
  metric, leveraging a scoring model fine-tuned with human-annotated data and
  systematic evaluation criteria, aimed at reducing bias and improving evaluation
  accuracy. Extensive experiments demonstrate the effectiveness of our benchmark
  and metrics in providing a comprehensive evaluation of interleaved LVLMs.
  Specifically, we evaluate eight LVLMs, revealing that even the best models show
  significant room for improvement, with most achieving only moderate results. We
  believe MMIE will drive further advancements in the development of interleaved
  LVLMs. We publicly release our benchmark and code in
  https://mmie-bench.github.io/.
paper_ai_summary: MMIE is a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models using diverse data and reliable automated evaluation metrics.
paper_ai_keywords: multimodal comprehension multimodal generation Large Vision-Language Models LVLMs automated evaluation metric multimodal queries human-annotated data systematic evaluation criteria