AutoStory: Generating Diverse Storytelling Images with Minimal Human
  Effort
Story visualization aims to generate a series of images that match the story
described in texts, and it requires the generated images to satisfy high
quality, alignment with the text description, and consistency in character
identities. Given the complexity of story visualization, existing methods
drastically simplify the problem by considering only a few specific characters
and scenarios, or requiring the users to provide per-image control conditions
such as sketches. However, these simplifications render these methods
incompetent for real applications. To this end, we propose an automated story
visualization system that can effectively generate diverse, high-quality, and
consistent sets of story images, with minimal human interactions. Specifically,
we utilize the comprehension and planning capabilities of large language models
for layout planning, and then leverage large-scale text-to-image models to
generate sophisticated story images based on the layout. We empirically find
that sparse control conditions, such as bounding boxes, are suitable for layout
planning, while dense control conditions, e.g., sketches and keypoints, are
suitable for generating high-quality image content. To obtain the best of both
worlds, we devise a dense condition generation module to transform simple
bounding box layouts into sketch or keypoint control conditions for final image
generation, which not only improves the image quality but also allows easy and
intuitive user interactions. In addition, we propose a simple yet effective
method to generate multi-view consistent character images, eliminating the
reliance on human labor to collect or draw character images.
655c245a0c27b3fb70822843
Wen Wang
claimed_verified
2023-11-21T13:50:46.620Z
63f089456309c84d5f47f951
/avatars/04b926a7f2ad091ee00fef0c59903492.svg
Wen Wang
user
wwen1997
655c245a0c27b3fb70822844
Canyu Zhao
admin_assigned
2023-11-21T13:35:56.105Z
646efd223dd912a539e0bd46
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
Canyu Zhao
user
Canyu
655c245a0c27b3fb70822845
Hao Chen
655c245a0c27b3fb70822846
Zhekai Chen
admin_assigned
2023-11-21T13:37:41.546Z
62d812e143df7719860d05d1
/avatars/412f7ec5c9f54990f4b562652d3e2c59.svg
zhekai chen
user
Azily
655c245a0c27b3fb70822847
Kecheng Zheng
extracted_confirmed
2024-03-26T10:22:19.846Z
64252045a4f3051f54dd1d53
/avatars/0e423a3291091be3b4736a14da3ce495.svg
kecheng zheng
user
zkcys001
655c245a0c27b3fb70822848
Chunhua Shen
AutoStory: Generating Diverse Storytelling Images with Minimal Human
  Effort
Story visualization aims to generate a series of images that match the story
described in texts, and it requires the generated images to satisfy high
quality, alignment with the text description, and consistency in character
identities. Given the complexity of story visualization, existing methods
drastically simplify the problem by considering only a few specific characters
and scenarios, or requiring the users to provide per-image control conditions
such as sketches. However, these simplifications render these methods
incompetent for real applications. To this end, we propose an automated story
visualization system that can effectively generate diverse, high-quality, and
consistent sets of story images, with minimal human interactions. Specifically,
we utilize the comprehension and planning capabilities of large language models
for layout planning, and then leverage large-scale text-to-image models to
generate sophisticated story images based on the layout. We empirically find
that sparse control conditions, such as bounding boxes, are suitable for layout
planning, while dense control conditions, e.g., sketches and keypoints, are
suitable for generating high-quality image content. To obtain the best of both
worlds, we devise a dense condition generation module to transform simple
bounding box layouts into sketch or keypoint control conditions for final image
generation, which not only improves the image quality but also allows easy and
intuitive user interactions. In addition, we propose a simple yet effective
method to generate multi-view consistent character images, eliminating the
reliance on human labor to collect or draw character images.
The system uses large language models for layout planning and large-scale text-to-image models for generating high-quality, consistent story images with minimal human interaction, transforming bounding boxes into dense control conditions to improve image quality.
large language models text-to-image models layout planning bounding boxes sketch keypoints multi-view consistent character images