title: Openstory++: A Large-scale Dataset and Benchmark for Instance-aware
    Open-domain Visual Storytelling
summary: Recent image generation models excel at creating high-quality images from
  brief captions. However, they fail to maintain consistency of multiple
  instances across images when encountering lengthy contexts. This inconsistency
  is largely due to in existing training datasets the absence of granular
  instance feature labeling in existing training datasets. To tackle these
  issues, we introduce Openstory++, a large-scale dataset combining additional
  instance-level annotations with both images and text. Furthermore, we develop a
  training methodology that emphasizes entity-centric image-text generation,
  ensuring that the models learn to effectively interweave visual and textual
  information. Specifically, Openstory++ streamlines the process of keyframe
  extraction from open-domain videos, employing vision-language models to
  generate captions that are then polished by a large language model for
  narrative continuity. It surpasses previous datasets by offering a more
  expansive open-domain resource, which incorporates automated captioning,
  high-resolution imagery tailored for instance count, and extensive frame
  sequences for temporal consistency. Additionally, we present Cohere-Bench, a
  pioneering benchmark framework for evaluating the image generation tasks when
  long multimodal context is provided, including the ability to keep the
  background, style, instances in the given context coherent. Compared to
  existing benchmarks, our work fills critical gaps in multi-modal generation,
  propelling the development of models that can adeptly generate and interpret
  complex narratives in open-domain environments. Experiments conducted within
  Cohere-Bench confirm the superiority of Openstory++ in nurturing high-quality
  visual storytelling models, enhancing their ability to address open-domain
  generation tasks. More details can be found at https://openstorypp.github.io/
paper_authors._id: 66b4252a8a1be1e8acc24e47
paper_authors.name: Zilyu Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:29:23.643Z
paper_authors._id: 65b62ab582d3845134ef0aab
paper_authors.avatarUrl: /avatars/fb1c22b2937e86f668b06fedb48e57e0.svg
paper_authors.fullname: Zilyu Ye
paper_authors.type: user
paper_authors.user: YeLuoSuiYou
paper_authors._id: 66b4252a8a1be1e8acc24e48
paper_authors.name: Jinxiu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:29:36.192Z
paper_authors._id: 6456785cbfdf9c63ce301bf2
paper_authors.avatarUrl: /avatars/db1b71e547d901cc6ed3c7f96a50b7b1.svg
paper_authors.fullname: Jinxiu
paper_authors.type: user
paper_authors.user: BrandonLiu
paper_authors._id: 66b4252a8a1be1e8acc24e49
paper_authors.name: Ruotian Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:29:42.839Z
paper_authors._id: 662f2c8435ab6df959b005de
paper_authors.avatarUrl: /avatars/3e30053ecbe9cc14b5e1eb2b014755de.svg
paper_authors.fullname: ruotian peng
paper_authors.type: user
paper_authors.user: prt66
paper_authors._id: 66b4252a8a1be1e8acc24e4a
paper_authors.name: Jinjin Cao
paper_authors._id: 66b4252a8a1be1e8acc24e4b
paper_authors.name: Zhiyang Chen
paper_authors._id: 66b4252a8a1be1e8acc24e4c
paper_authors.name: Yiyang Zhang
paper_authors._id: 66b4252a8a1be1e8acc24e4d
paper_authors.name: Ziwei Xuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:31:14.079Z
paper_authors._id: 646fa2bc799a974be31a3158
paper_authors.avatarUrl: /avatars/d48453bf9908305765728f9106a4f96a.svg
paper_authors.fullname: Ziwei Xuan
paper_authors.type: user
paper_authors.user: gokyo
paper_authors._id: 66b4252a8a1be1e8acc24e4e
paper_authors.name: Mingyuan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:31:06.376Z
paper_authors._id: 653b2524b77b5e255f2d29d2
paper_authors.avatarUrl: /avatars/f69aea8de84c435295e7638bad5bd82e.svg
paper_authors.fullname: Mingyuan Zhou
paper_authors.type: user
paper_authors.user: mingyuanzhou
paper_authors._id: 66b4252a8a1be1e8acc24e4f
paper_authors.name: Xiaoqian Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:30:59.750Z
paper_authors._id: 620c5b20c33ca69c4c82145b
paper_authors.avatarUrl: /avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
paper_authors.fullname: Xiaoqian Shen
paper_authors.type: user
paper_authors.user: shenxq
paper_authors._id: 66b4252a8a1be1e8acc24e50
paper_authors.name: Mohamed Elhoseiny
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-08T10:30:53.056Z
paper_authors._id: 64a27d39649b0d08ca0a4ca6
paper_authors.avatarUrl: /avatars/e4446a875506c10de9ae28411dc6416d.svg
paper_authors.fullname: Mohamed Elhoseiny
paper_authors.type: user
paper_authors.user: mhelhoseiny
paper_authors._id: 66b4252a8a1be1e8acc24e51
paper_authors.name: Qi Liu
paper_authors._id: 66b4252a8a1be1e8acc24e52
paper_authors.name: Guo-Jun Qi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-08T01:53:52.986Z
paper_authors._id: 665449b3574a7aab99939880
paper_authors.avatarUrl: /avatars/df916228b2231e7a14d55bdd6556eebf.svg
paper_authors.fullname: Guo-Jun Qi
paper_authors.type: user
paper_authors.user: guojunq
paper_title: Openstory++: A Large-scale Dataset and Benchmark for Instance-aware
    Open-domain Visual Storytelling
paper_summary: Recent image generation models excel at creating high-quality images from
  brief captions. However, they fail to maintain consistency of multiple
  instances across images when encountering lengthy contexts. This inconsistency
  is largely due to in existing training datasets the absence of granular
  instance feature labeling in existing training datasets. To tackle these
  issues, we introduce Openstory++, a large-scale dataset combining additional
  instance-level annotations with both images and text. Furthermore, we develop a
  training methodology that emphasizes entity-centric image-text generation,
  ensuring that the models learn to effectively interweave visual and textual
  information. Specifically, Openstory++ streamlines the process of keyframe
  extraction from open-domain videos, employing vision-language models to
  generate captions that are then polished by a large language model for
  narrative continuity. It surpasses previous datasets by offering a more
  expansive open-domain resource, which incorporates automated captioning,
  high-resolution imagery tailored for instance count, and extensive frame
  sequences for temporal consistency. Additionally, we present Cohere-Bench, a
  pioneering benchmark framework for evaluating the image generation tasks when
  long multimodal context is provided, including the ability to keep the
  background, style, instances in the given context coherent. Compared to
  existing benchmarks, our work fills critical gaps in multi-modal generation,
  propelling the development of models that can adeptly generate and interpret
  complex narratives in open-domain environments. Experiments conducted within
  Cohere-Bench confirm the superiority of Openstory++ in nurturing high-quality
  visual storytelling models, enhancing their ability to address open-domain
  generation tasks. More details can be found at https://openstorypp.github.io/
paper_ai_keywords: entities-centric instance-level annotations keyframe vision-vox models large language model narrative continuity automated captioning high-resolution imagery frame sequences temporal consistency Cohere-Bench multimodal tasks visual storytelling models open-domain generation tasks