title: AutoStory: Generating Diverse Storytelling Images with Minimal Human
    Effort
summary: Story visualization aims to generate a series of images that match the story
  described in texts, and it requires the generated images to satisfy high
  quality, alignment with the text description, and consistency in character
  identities. Given the complexity of story visualization, existing methods
  drastically simplify the problem by considering only a few specific characters
  and scenarios, or requiring the users to provide per-image control conditions
  such as sketches. However, these simplifications render these methods
  incompetent for real applications. To this end, we propose an automated story
  visualization system that can effectively generate diverse, high-quality, and
  consistent sets of story images, with minimal human interactions. Specifically,
  we utilize the comprehension and planning capabilities of large language models
  for layout planning, and then leverage large-scale text-to-image models to
  generate sophisticated story images based on the layout. We empirically find
  that sparse control conditions, such as bounding boxes, are suitable for layout
  planning, while dense control conditions, e.g., sketches and keypoints, are
  suitable for generating high-quality image content. To obtain the best of both
  worlds, we devise a dense condition generation module to transform simple
  bounding box layouts into sketch or keypoint control conditions for final image
  generation, which not only improves the image quality but also allows easy and
  intuitive user interactions. In addition, we propose a simple yet effective
  method to generate multi-view consistent character images, eliminating the
  reliance on human labor to collect or draw character images.
paper_authors._id: 655c245a0c27b3fb70822843
paper_authors.name: Wen Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-21T13:50:46.620Z
paper_authors._id: 63f089456309c84d5f47f951
paper_authors.avatarUrl: /avatars/04b926a7f2ad091ee00fef0c59903492.svg
paper_authors.fullname: Wen Wang
paper_authors.type: user
paper_authors.user: wwen1997
paper_authors._id: 655c245a0c27b3fb70822844
paper_authors.name: Canyu Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:35:56.105Z
paper_authors._id: 646efd223dd912a539e0bd46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EOFAv5xvOgJOzuDgh4nSb.png
paper_authors.fullname: Canyu Zhao
paper_authors.type: user
paper_authors.user: Canyu
paper_authors._id: 655c245a0c27b3fb70822845
paper_authors.name: Hao Chen
paper_authors._id: 655c245a0c27b3fb70822846
paper_authors.name: Zhekai Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T13:37:41.546Z
paper_authors._id: 62d812e143df7719860d05d1
paper_authors.avatarUrl: /avatars/412f7ec5c9f54990f4b562652d3e2c59.svg
paper_authors.fullname: zhekai chen
paper_authors.type: user
paper_authors.user: Azily
paper_authors._id: 655c245a0c27b3fb70822847
paper_authors.name: Kecheng Zheng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-26T10:22:19.846Z
paper_authors._id: 64252045a4f3051f54dd1d53
paper_authors.avatarUrl: /avatars/0e423a3291091be3b4736a14da3ce495.svg
paper_authors.fullname: kecheng zheng
paper_authors.type: user
paper_authors.user: zkcys001
paper_authors._id: 655c245a0c27b3fb70822848
paper_authors.name: Chunhua Shen
paper_title: AutoStory: Generating Diverse Storytelling Images with Minimal Human
    Effort
paper_summary: Story visualization aims to generate a series of images that match the story
  described in texts, and it requires the generated images to satisfy high
  quality, alignment with the text description, and consistency in character
  identities. Given the complexity of story visualization, existing methods
  drastically simplify the problem by considering only a few specific characters
  and scenarios, or requiring the users to provide per-image control conditions
  such as sketches. However, these simplifications render these methods
  incompetent for real applications. To this end, we propose an automated story
  visualization system that can effectively generate diverse, high-quality, and
  consistent sets of story images, with minimal human interactions. Specifically,
  we utilize the comprehension and planning capabilities of large language models
  for layout planning, and then leverage large-scale text-to-image models to
  generate sophisticated story images based on the layout. We empirically find
  that sparse control conditions, such as bounding boxes, are suitable for layout
  planning, while dense control conditions, e.g., sketches and keypoints, are
  suitable for generating high-quality image content. To obtain the best of both
  worlds, we devise a dense condition generation module to transform simple
  bounding box layouts into sketch or keypoint control conditions for final image
  generation, which not only improves the image quality but also allows easy and
  intuitive user interactions. In addition, we propose a simple yet effective
  method to generate multi-view consistent character images, eliminating the
  reliance on human labor to collect or draw character images.
paper_ai_summary: The system uses large language models for layout planning and large-scale text-to-image models for generating high-quality, consistent story images with minimal human interaction, transforming bounding boxes into dense control conditions to improve image quality.
paper_ai_keywords: large language models text-to-image models layout planning bounding boxes sketch keypoints multi-view consistent character images