MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large
  Vision-Language Models
Interleaved multimodal comprehension and generation, enabling models to
produce and interpret both images and text in arbitrary sequences, have become
a pivotal area in multimodal learning. Despite significant advancements, the
evaluation of this capability remains insufficient. Existing benchmarks suffer
from limitations in data scale, scope, and evaluation depth, while current
evaluation metrics are often costly or biased, lacking in reliability for
practical applications. To address these challenges, we introduce MMIE, a
large-scale knowledge-intensive benchmark for evaluating interleaved multimodal
comprehension and generation in Large Vision-Language Models (LVLMs). MMIE
comprises 20K meticulously curated multimodal queries, spanning 3 categories,
12 fields, and 102 subfields, including mathematics, coding, physics,
literature, health, and arts. It supports both interleaved inputs and outputs,
offering a mix of multiple-choice and open-ended question formats to evaluate
diverse competencies. Moreover, we propose a reliable automated evaluation
metric, leveraging a scoring model fine-tuned with human-annotated data and
systematic evaluation criteria, aimed at reducing bias and improving evaluation
accuracy. Extensive experiments demonstrate the effectiveness of our benchmark
and metrics in providing a comprehensive evaluation of interleaved LVLMs.
Specifically, we evaluate eight LVLMs, revealing that even the best models show
significant room for improvement, with most achieving only moderate results. We
believe MMIE will drive further advancements in the development of interleaved
LVLMs. We publicly release our benchmark and code in
https://mmie-bench.github.io/.
670dc9101fd338212b902556
Peng Xia
claimed_verified
2024-10-15T09:11:03.755Z
643e9ee6f6bb3c31a26e7bc4
/avatars/acfaa7d6a23dada24c86b954c3be116a.svg
Peng Xia
user
richardxp888
670dc9101fd338212b902557
Siwei Han
admin_assigned
2024-10-15T08:37:39.675Z
65941852f0152a21fc860f79
https://cdn-avatars.huggingface.co/v1/production/uploads/65941852f0152a21fc860f79/mHOkrMOkivxxK__F4AK2o.png
Lillianwei
user
Lillianwei
670dc9101fd338212b902558
Shi Qiu
claimed_verified
2024-10-15T09:11:02.150Z
6644bb2c9bdbd85493074411
https://cdn-avatars.huggingface.co/v1/production/uploads/6644bb2c9bdbd85493074411/iug8dkP1zjID-kXzNxDjD.jpeg
SHI QIU
user
StarThomas1002
670dc9101fd338212b902559
Yiyang Zhou
670dc9101fd338212b90255a
Zhaoyang Wang
admin_assigned
2024-10-15T09:26:59.440Z
66c719e2f3f9994f2ae6c1d2
/avatars/c6cbbf458e30d235bc3db12628510fc4.svg
zhaoyang.wang
user
wangzhaoyang
670dc9101fd338212b90255b
Wenhao Zheng
admin_assigned
2024-10-15T09:27:06.091Z
65d0c9b4fff50114959c0004
/avatars/31f599204e51528ec317aeb980f4676e.svg
Wenhao Zheng
user
WendellZwh
670dc9101fd338212b90255c
Zhaorun Chen
admin_assigned
2024-10-15T09:27:12.112Z
64d448d3d095077728f7c740
https://cdn-avatars.huggingface.co/v1/production/uploads/64d448d3d095077728f7c740/4CFkiVoXXDsu7uQPTU0n3.jpeg
Zhaorun Chen
user
Zhaorun
670dc9101fd338212b90255d
Chenhang Cui
admin_assigned
2024-10-15T09:27:18.302Z
64a26627f09ad861c85487e0
/avatars/baefd153c59f3a3238fd6034c6daf0d4.svg
ChenhangCui
user
Chenhangcui
670dc9101fd338212b90255e
Mingyu Ding
670dc9101fd338212b90255f
Linjie Li
admin_assigned
2024-10-15T09:27:58.086Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
670dc9101fd338212b902560
Lijuan Wang
admin_assigned
2024-10-15T09:28:14.317Z
6672e20d1dbdf7da8310dd92
/avatars/5d2fb23f92a7f9ff025a5be17a26de4d.svg
lijuanwang
user
lijuanwang228
670dc9101fd338212b902561
Huaxiu Yao
admin_assigned
2024-10-15T09:28:27.413Z
65f667a455009c4ad9e6ac4c
/avatars/ed10c2cf2ba3fde6d7da93f076961607.svg
Yao
user
Huaxiu
MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large
  Vision-Language Models
Interleaved multimodal comprehension and generation, enabling models to
produce and interpret both images and text in arbitrary sequences, have become
a pivotal area in multimodal learning. Despite significant advancements, the
evaluation of this capability remains insufficient. Existing benchmarks suffer
from limitations in data scale, scope, and evaluation depth, while current
evaluation metrics are often costly or biased, lacking in reliability for
practical applications. To address these challenges, we introduce MMIE, a
large-scale knowledge-intensive benchmark for evaluating interleaved multimodal
comprehension and generation in Large Vision-Language Models (LVLMs). MMIE
comprises 20K meticulously curated multimodal queries, spanning 3 categories,
12 fields, and 102 subfields, including mathematics, coding, physics,
literature, health, and arts. It supports both interleaved inputs and outputs,
offering a mix of multiple-choice and open-ended question formats to evaluate
diverse competencies. Moreover, we propose a reliable automated evaluation
metric, leveraging a scoring model fine-tuned with human-annotated data and
systematic evaluation criteria, aimed at reducing bias and improving evaluation
accuracy. Extensive experiments demonstrate the effectiveness of our benchmark
and metrics in providing a comprehensive evaluation of interleaved LVLMs.
Specifically, we evaluate eight LVLMs, revealing that even the best models show
significant room for improvement, with most achieving only moderate results. We
believe MMIE will drive further advancements in the development of interleaved
LVLMs. We publicly release our benchmark and code in
https://mmie-bench.github.io/.
MMIE is a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models using diverse data and reliable automated evaluation metrics.
multimodal comprehension multimodal generation Large Vision-Language Models LVLMs automated evaluation metric multimodal queries human-annotated data systematic evaluation criteria