Openstory++: A Large-scale Dataset and Benchmark for Instance-aware
  Open-domain Visual Storytelling
Recent image generation models excel at creating high-quality images from
brief captions. However, they fail to maintain consistency of multiple
instances across images when encountering lengthy contexts. This inconsistency
is largely due to in existing training datasets the absence of granular
instance feature labeling in existing training datasets. To tackle these
issues, we introduce Openstory++, a large-scale dataset combining additional
instance-level annotations with both images and text. Furthermore, we develop a
training methodology that emphasizes entity-centric image-text generation,
ensuring that the models learn to effectively interweave visual and textual
information. Specifically, Openstory++ streamlines the process of keyframe
extraction from open-domain videos, employing vision-language models to
generate captions that are then polished by a large language model for
narrative continuity. It surpasses previous datasets by offering a more
expansive open-domain resource, which incorporates automated captioning,
high-resolution imagery tailored for instance count, and extensive frame
sequences for temporal consistency. Additionally, we present Cohere-Bench, a
pioneering benchmark framework for evaluating the image generation tasks when
long multimodal context is provided, including the ability to keep the
background, style, instances in the given context coherent. Compared to
existing benchmarks, our work fills critical gaps in multi-modal generation,
propelling the development of models that can adeptly generate and interpret
complex narratives in open-domain environments. Experiments conducted within
Cohere-Bench confirm the superiority of Openstory++ in nurturing high-quality
visual storytelling models, enhancing their ability to address open-domain
generation tasks. More details can be found at https://openstorypp.github.io/
66b4252a8a1be1e8acc24e47
Zilyu Ye
admin_assigned
2024-08-08T10:29:23.643Z
65b62ab582d3845134ef0aab
/avatars/fb1c22b2937e86f668b06fedb48e57e0.svg
Zilyu Ye
user
YeLuoSuiYou
66b4252a8a1be1e8acc24e48
Jinxiu Liu
admin_assigned
2024-08-08T10:29:36.192Z
6456785cbfdf9c63ce301bf2
/avatars/db1b71e547d901cc6ed3c7f96a50b7b1.svg
Jinxiu
user
BrandonLiu
66b4252a8a1be1e8acc24e49
Ruotian Peng
admin_assigned
2024-08-08T10:29:42.839Z
662f2c8435ab6df959b005de
/avatars/3e30053ecbe9cc14b5e1eb2b014755de.svg
ruotian peng
user
prt66
66b4252a8a1be1e8acc24e4a
Jinjin Cao
66b4252a8a1be1e8acc24e4b
Zhiyang Chen
66b4252a8a1be1e8acc24e4c
Yiyang Zhang
66b4252a8a1be1e8acc24e4d
Ziwei Xuan
admin_assigned
2024-08-08T10:31:14.079Z
646fa2bc799a974be31a3158
/avatars/d48453bf9908305765728f9106a4f96a.svg
Ziwei Xuan
user
gokyo
66b4252a8a1be1e8acc24e4e
Mingyuan Zhou
admin_assigned
2024-08-08T10:31:06.376Z
653b2524b77b5e255f2d29d2
/avatars/f69aea8de84c435295e7638bad5bd82e.svg
Mingyuan Zhou
user
mingyuanzhou
66b4252a8a1be1e8acc24e4f
Xiaoqian Shen
admin_assigned
2024-08-08T10:30:59.750Z
620c5b20c33ca69c4c82145b
/avatars/f3b532c47e434be30d1005fe6d0e25fb.svg
Xiaoqian Shen
user
shenxq
66b4252a8a1be1e8acc24e50
Mohamed Elhoseiny
admin_assigned
2024-08-08T10:30:53.056Z
64a27d39649b0d08ca0a4ca6
/avatars/e4446a875506c10de9ae28411dc6416d.svg
Mohamed Elhoseiny
user
mhelhoseiny
66b4252a8a1be1e8acc24e51
Qi Liu
66b4252a8a1be1e8acc24e52
Guo-Jun Qi
extracted_pending
2024-08-08T01:53:52.986Z
665449b3574a7aab99939880
/avatars/df916228b2231e7a14d55bdd6556eebf.svg
Guo-Jun Qi
user
guojunq
Openstory++: A Large-scale Dataset and Benchmark for Instance-aware
  Open-domain Visual Storytelling
Recent image generation models excel at creating high-quality images from
brief captions. However, they fail to maintain consistency of multiple
instances across images when encountering lengthy contexts. This inconsistency
is largely due to in existing training datasets the absence of granular
instance feature labeling in existing training datasets. To tackle these
issues, we introduce Openstory++, a large-scale dataset combining additional
instance-level annotations with both images and text. Furthermore, we develop a
training methodology that emphasizes entity-centric image-text generation,
ensuring that the models learn to effectively interweave visual and textual
information. Specifically, Openstory++ streamlines the process of keyframe
extraction from open-domain videos, employing vision-language models to
generate captions that are then polished by a large language model for
narrative continuity. It surpasses previous datasets by offering a more
expansive open-domain resource, which incorporates automated captioning,
high-resolution imagery tailored for instance count, and extensive frame
sequences for temporal consistency. Additionally, we present Cohere-Bench, a
pioneering benchmark framework for evaluating the image generation tasks when
long multimodal context is provided, including the ability to keep the
background, style, instances in the given context coherent. Compared to
existing benchmarks, our work fills critical gaps in multi-modal generation,
propelling the development of models that can adeptly generate and interpret
complex narratives in open-domain environments. Experiments conducted within
Cohere-Bench confirm the superiority of Openstory++ in nurturing high-quality
visual storytelling models, enhancing their ability to address open-domain
generation tasks. More details can be found at https://openstorypp.github.io/
A new dataset Openstory++ and benchmark Cohere-Bench improve image generation by enhancing instance feature labeling and evaluating consistency in long multimodal contexts.
vision-language models large language model keyframe extraction captioning high-resolution imagery frame sequences multimodal generation benchmark framework visual storytelling open-domain generation