GATE OpenING: A Comprehensive Benchmark for Judging Open-ended
  Interleaved Image-Text Generation
Multimodal Large Language Models (MLLMs) have made significant strides in
visual understanding and generation tasks. However, generating interleaved
image-text content remains a challenge, which requires integrated multimodal
understanding and generation abilities. While the progress in unified models
offers new solutions, existing benchmarks are insufficient for evaluating these
methods due to data size and diversity limitations. To bridge this gap, we
introduce GATE OpenING (OpenING), a comprehensive benchmark comprising 5,400
high-quality human-annotated instances across 56 real-world tasks. OpenING
covers diverse daily scenarios such as travel guide, design, and brainstorming,
offering a robust platform for challenging interleaved generation methods. In
addition, we present IntJudge, a judge model for evaluating open-ended
multimodal generation methods. Trained with a novel data pipeline, our IntJudge
achieves an agreement rate of 82. 42% with human judgments, outperforming
GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that
current interleaved generation methods still have substantial room for
improvement. Key findings on interleaved image-text generation are further
presented to guide the development of next-generation models. The OpenING is
open-sourced at https://opening.github.io.
6749af6061f0c91e2ea76a1e
Pengfei Zhou
admin_assigned
2024-12-03T11:29:04.250Z
65df481e530333731ea24617
/avatars/3ed41f5d7d0489193807b5e6260f16c9.svg
ZHOU PENGFEI
user
lyuukuu
6749af6061f0c91e2ea76a1f
Xiaopeng Peng
claimed_verified
2024-12-05T09:29:46.356Z
66c3ea15a545b0cefec55333
https://cdn-avatars.huggingface.co/v1/production/uploads/66c3ea15a545b0cefec55333/sHLahe_zZhXE450vXFLUv.jpeg
Xiaopeng Peng
user
callmescarlett
6749af6061f0c91e2ea76a20
Jiajun Song
6749af6061f0c91e2ea76a21
Chuanhao Li
claimed_verified
2025-07-17T13:56:49.281Z
674ea59a8f2e7614a6c72f26
/avatars/86fd4c6d7d33435de49b35659bf65265.svg
Chuanhao
user
ChuanhaoLi
6749af6061f0c91e2ea76a22
Zhaopan Xu
admin_assigned
2024-12-03T12:00:31.371Z
646dee28174cc96d50951991
/avatars/17d88a24c905e9819268b27037015a35.svg
xu
user
xuzhaopan
6749af6061f0c91e2ea76a23
Yue Yang
6749af6061f0c91e2ea76a24
Ziyao Guo
6749af6061f0c91e2ea76a25
Hao Zhang
6749af6061f0c91e2ea76a26
Yuqi Lin
6749af6061f0c91e2ea76a27
Yefei He
admin_assigned
2024-12-03T11:30:18.425Z
65a88c3d26598b995531fff1
/avatars/b1a524857d8572d0405476661b434160.svg
Yefei He
user
yefly
6749af6061f0c91e2ea76a28
Lirui Zhao
admin_assigned
2024-12-03T11:30:12.376Z
661639c4dd9284ce5d361f0f
/avatars/9a0f8f725d7b90c30dfdb64313761155.svg
Lirui Zhao
user
LiruiZhao
6749af6061f0c91e2ea76a29
Shuo Liu
6749af6061f0c91e2ea76a2a
Tianhua Li
6749af6061f0c91e2ea76a2b
Yuxuan Xie
6749af6061f0c91e2ea76a2c
Xiaojun Chang
6749af6061f0c91e2ea76a2d
Yu Qiao
6749af6061f0c91e2ea76a2e
Wenqi Shao
admin_assigned
2024-12-03T11:29:33.418Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
6749af6061f0c91e2ea76a2f
Kaipeng Zhang
admin_assigned
2024-12-03T11:29:27.230Z
63527f4e7d071f23d085ad45
/avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
KAIPENG ZHANG
user
kpzhang
GATE OpenING: A Comprehensive Benchmark for Judging Open-ended
  Interleaved Image-Text Generation
Multimodal Large Language Models (MLLMs) have made significant strides in
visual understanding and generation tasks. However, generating interleaved
image-text content remains a challenge, which requires integrated multimodal
understanding and generation abilities. While the progress in unified models
offers new solutions, existing benchmarks are insufficient for evaluating these
methods due to data size and diversity limitations. To bridge this gap, we
introduce GATE OpenING (OpenING), a comprehensive benchmark comprising 5,400
high-quality human-annotated instances across 56 real-world tasks. OpenING
covers diverse daily scenarios such as travel guide, design, and brainstorming,
offering a robust platform for challenging interleaved generation methods. In
addition, we present IntJudge, a judge model for evaluating open-ended
multimodal generation methods. Trained with a novel data pipeline, our IntJudge
achieves an agreement rate of 82. 42% with human judgments, outperforming
GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that
current interleaved generation methods still have substantial room for
improvement. Key findings on interleaved image-text generation are further
presented to guide the development of next-generation models. The OpenING is
open-sourced at https://opening.github.io.
GATE OpenING is a benchmark with 5,400 instances across 56 tasks for evaluating interleaved image-text generation, complemented by an evaluative model IntJudge that outperforms GPT-based models.
Multimodal Large Language Models MLLMs visual understanding image-text content unified models benchmark human-annotated instances daily scenarios IntJudge judge model open-ended multimodal generation data pipeline agreement rate interleaved generation next-generation models