title: GATE OpenING: A Comprehensive Benchmark for Judging Open-ended
    Interleaved Image-Text Generation
summary: Multimodal Large Language Models (MLLMs) have made significant strides in
  visual understanding and generation tasks. However, generating interleaved
  image-text content remains a challenge, which requires integrated multimodal
  understanding and generation abilities. While the progress in unified models
  offers new solutions, existing benchmarks are insufficient for evaluating these
  methods due to data size and diversity limitations. To bridge this gap, we
  introduce GATE OpenING (OpenING), a comprehensive benchmark comprising 5,400
  high-quality human-annotated instances across 56 real-world tasks. OpenING
  covers diverse daily scenarios such as travel guide, design, and brainstorming,
  offering a robust platform for challenging interleaved generation methods. In
  addition, we present IntJudge, a judge model for evaluating open-ended
  multimodal generation methods. Trained with a novel data pipeline, our IntJudge
  achieves an agreement rate of 82. 42% with human judgments, outperforming
  GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that
  current interleaved generation methods still have substantial room for
  improvement. Key findings on interleaved image-text generation are further
  presented to guide the development of next-generation models. The OpenING is
  open-sourced at https://opening.github.io.
paper_authors._id: 6749af6061f0c91e2ea76a1e
paper_authors.name: Pengfei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T11:29:04.250Z
paper_authors._id: 65df481e530333731ea24617
paper_authors.avatarUrl: /avatars/3ed41f5d7d0489193807b5e6260f16c9.svg
paper_authors.fullname: ZHOU PENGFEI
paper_authors.type: user
paper_authors.user: lyuukuu
paper_authors._id: 6749af6061f0c91e2ea76a1f
paper_authors.name: Xiaopeng Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:29:46.356Z
paper_authors._id: 66c3ea15a545b0cefec55333
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66c3ea15a545b0cefec55333/Uifod2dtTDrPw3afPr6Qd.jpeg
paper_authors.fullname: Xiaopeng Peng
paper_authors.type: user
paper_authors.user: callmescarlett
paper_authors._id: 6749af6061f0c91e2ea76a20
paper_authors.name: Jiajun Song
paper_authors._id: 6749af6061f0c91e2ea76a21
paper_authors.name: Chuanhao Li
paper_authors._id: 6749af6061f0c91e2ea76a22
paper_authors.name: Zhaopan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T12:00:31.371Z
paper_authors._id: 646dee28174cc96d50951991
paper_authors.avatarUrl: /avatars/17d88a24c905e9819268b27037015a35.svg
paper_authors.fullname: xu
paper_authors.type: user
paper_authors.user: xuzhaopan
paper_authors._id: 6749af6061f0c91e2ea76a23
paper_authors.name: Yue Yang
paper_authors._id: 6749af6061f0c91e2ea76a24
paper_authors.name: Ziyao Guo
paper_authors._id: 6749af6061f0c91e2ea76a25
paper_authors.name: Hao Zhang
paper_authors._id: 6749af6061f0c91e2ea76a26
paper_authors.name: Yuqi Lin
paper_authors._id: 6749af6061f0c91e2ea76a27
paper_authors.name: Yefei He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T11:30:18.425Z
paper_authors._id: 65a88c3d26598b995531fff1
paper_authors.avatarUrl: /avatars/b1a524857d8572d0405476661b434160.svg
paper_authors.fullname: Yefei He
paper_authors.type: user
paper_authors.user: yefly
paper_authors._id: 6749af6061f0c91e2ea76a28
paper_authors.name: Lirui Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T11:30:12.376Z
paper_authors._id: 661639c4dd9284ce5d361f0f
paper_authors.avatarUrl: /avatars/9a0f8f725d7b90c30dfdb64313761155.svg
paper_authors.fullname: Lirui Zhao
paper_authors.type: user
paper_authors.user: LiruiZhao
paper_authors._id: 6749af6061f0c91e2ea76a29
paper_authors.name: Shuo Liu
paper_authors._id: 6749af6061f0c91e2ea76a2a
paper_authors.name: Tianhua Li
paper_authors._id: 6749af6061f0c91e2ea76a2b
paper_authors.name: Yuxuan Xie
paper_authors._id: 6749af6061f0c91e2ea76a2c
paper_authors.name: Xiaojun Chang
paper_authors._id: 6749af6061f0c91e2ea76a2d
paper_authors.name: Yu Qiao
paper_authors._id: 6749af6061f0c91e2ea76a2e
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T11:29:33.418Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 6749af6061f0c91e2ea76a2f
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T11:29:27.230Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_title: GATE OpenING: A Comprehensive Benchmark for Judging Open-ended
    Interleaved Image-Text Generation
paper_summary: Multimodal Large Language Models (MLLMs) have made significant strides in
  visual understanding and generation tasks. However, generating interleaved
  image-text content remains a challenge, which requires integrated multimodal
  understanding and generation abilities. While the progress in unified models
  offers new solutions, existing benchmarks are insufficient for evaluating these
  methods due to data size and diversity limitations. To bridge this gap, we
  introduce GATE OpenING (OpenING), a comprehensive benchmark comprising 5,400
  high-quality human-annotated instances across 56 real-world tasks. OpenING
  covers diverse daily scenarios such as travel guide, design, and brainstorming,
  offering a robust platform for challenging interleaved generation methods. In
  addition, we present IntJudge, a judge model for evaluating open-ended
  multimodal generation methods. Trained with a novel data pipeline, our IntJudge
  achieves an agreement rate of 82. 42% with human judgments, outperforming
  GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that
  current interleaved generation methods still have substantial room for
  improvement. Key findings on interleaved image-text generation are further
  presented to guide the development of next-generation models. The OpenING is
  open-sourced at https://opening.github.io.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) visual understanding generation tasks interleaved image-text content unified models benchmark human-annotated instances real-world tasks daily scenarios GATE OpenING (OpenING) CL IntJudge open-ended multimodal generation methods data pipeline agreement rate GPT-based evaluators interleaved image-text generation next-generation models