title: JourneyDB: A Benchmark for Generative Image Understanding
summary: While recent advancements in vision-language models have revolutionized
  multi-modal understanding, it remains unclear whether they possess the
  capabilities of comprehending the generated images. Compared to real data,
  synthetic images exhibit a higher degree of diversity in both content and
  style, for which there are significant difficulties for the models to fully
  apprehend. To this end, we present a large-scale dataset, JourneyDB, for
  multi-modal visual understanding in generative images. Our curated dataset
  covers 4 million diverse and high-quality generated images paired with the text
  prompts used to produce them. We further design 4 benchmarks to quantify the
  performance of generated image understanding in terms of both content and style
  interpretation. These benchmarks include prompt inversion, style retrieval,
  image captioning and visual question answering. Lastly, we assess the
  performance of current state-of-the-art multi-modal models when applied to
  JourneyDB, and provide an in-depth analysis of their strengths and limitations
  in generated content understanding. We hope the proposed dataset and benchmarks
  will facilitate the research in the field of generative content understanding.
  The dataset will be available on https://journeydb.github.io.
paper_authors._id: 64a384400d5af4b16211ee2b
paper_authors.name: Junting Pan
paper_authors._id: 64a384400d5af4b16211ee2c
paper_authors.name: Keqiang Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:55:55.200Z
paper_authors._id: 645ca9f3a19f3e64bbed7534
paper_authors.avatarUrl: /avatars/37fde5fd978bc03a3a02c30102557432.svg
paper_authors.fullname: Keqiang Sun
paper_authors.type: user
paper_authors.user: kqsun
paper_authors._id: 64a384400d5af4b16211ee2d
paper_authors.name: Yuying Ge
paper_authors._id: 64a384400d5af4b16211ee2e
paper_authors.name: Hao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:44:44.723Z
paper_authors._id: 64b9033777ae61bcc80aa4f3
paper_authors.avatarUrl: /avatars/408c335395c79f3df69fd9bf70abc312.svg
paper_authors.fullname: Hao Li
paper_authors.type: user
paper_authors.user: cpsxhao
paper_authors._id: 64a384400d5af4b16211ee2f
paper_authors.name: Haodong Duan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T12:50:36.834Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 64a384400d5af4b16211ee30
paper_authors.name: Xiaoshi Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T19:17:00.843Z
paper_authors._id: 6423f44d30b0e4ab36dd939d
paper_authors.avatarUrl: /avatars/9bbf6f00188d21d7bd5b678b955b9fc0.svg
paper_authors.fullname: Xiaoshi Wu
paper_authors.type: user
paper_authors.user: xswu
paper_authors._id: 64a384400d5af4b16211ee31
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:40:57.242Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 64a384400d5af4b16211ee32
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T10:06:08.713Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 64a384400d5af4b16211ee33
paper_authors.name: Zipeng Qin
paper_authors._id: 64a384400d5af4b16211ee34
paper_authors.name: Yi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-05T18:56:08.991Z
paper_authors._id: 63119a987680dc699b2031df
paper_authors.avatarUrl: /avatars/db63ab73806ee9cb8aa01be82d9effdd.svg
paper_authors.fullname: Yi Wang
paper_authors.type: user
paper_authors.user: shepnerd
paper_authors._id: 64a384400d5af4b16211ee35
paper_authors.name: Jifeng Dai
paper_authors._id: 64a384400d5af4b16211ee36
paper_authors.name: Yu Qiao
paper_authors._id: 64a384400d5af4b16211ee37
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:50:51.845Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: JourneyDB: A Benchmark for Generative Image Understanding
paper_summary: While recent advancements in vision-language models have revolutionized
  multi-modal understanding, it remains unclear whether they possess the
  capabilities of comprehending the generated images. Compared to real data,
  synthetic images exhibit a higher degree of diversity in both content and
  style, for which there are significant difficulties for the models to fully
  apprehend. To this end, we present a large-scale dataset, JourneyDB, for
  multi-modal visual understanding in generative images. Our curated dataset
  covers 4 million diverse and high-quality generated images paired with the text
  prompts used to produce them. We further design 4 benchmarks to quantify the
  performance of generated image understanding in terms of both content and style
  interpretation. These benchmarks include prompt inversion, style retrieval,
  image captioning and visual question answering. Lastly, we assess the
  performance of current state-of-the-art multi-modal models when applied to
  JourneyDB, and provide an in-depth analysis of their strengths and limitations
  in generated content understanding. We hope the proposed dataset and benchmarks
  will facilitate the research in the field of generative content understanding.
  The dataset will be available on https://journeydb.github.io.
paper_ai_keywords: vision-language models multi-modal understanding generative images JourneyDB prompt inversion style retrieval image captioning visual question answering generated content understanding