title: TaleCrafter: Interactive Story Visualization with Multiple Characters
summary: Accurate Story visualization requires several necessary elements, such as
  identity consistency across frames, the alignment between plain text and visual
  content, and a reasonable layout of objects in images. Most previous works
  endeavor to meet these requirements by fitting a text-to-image (T2I) model on a
  set of videos in the same style and with the same characters, e.g., the
  FlintstonesSV dataset. However, the learned T2I models typically struggle to
  adapt to new characters, scenes, and styles, and often lack the flexibility to
  revise the layout of the synthesized images. This paper proposes a system for
  generic interactive story visualization, capable of handling multiple novel
  characters and supporting the editing of layout and local structure. It is
  developed by leveraging the prior knowledge of large language and T2I models,
  trained on massive corpora. The system comprises four interconnected
  components: story-to-prompt generation (S2P), text-to-layout generation (T2L),
  controllable text-to-image generation (C-T2I), and image-to-video animation
  (I2V). First, the S2P module converts concise story information into detailed
  prompts required for subsequent stages. Next, T2L generates diverse and
  reasonable layouts based on the prompts, offering users the ability to adjust
  and refine the layout to their preference. The core component, C-T2I, enables
  the creation of images guided by layouts, sketches, and actor-specific
  identifiers to maintain consistency and detail across visualizations. Finally,
  I2V enriches the visualization process by animating the generated images.
  Extensive experiments and a user study are conducted to validate the
  effectiveness and flexibility of interactive editing of the proposed system.
paper_authors._id: 64757cbebb0e9dd7764b47cd
paper_authors.name: Yuan Gong
paper_authors._id: 64757cbebb0e9dd7764b47ce
paper_authors.name: Youxin Pang
paper_authors._id: 64757cbebb0e9dd7764b47cf
paper_authors.name: Xiaodong Cun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T07:46:38.942Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 64757cbebb0e9dd7764b47d0
paper_authors.name: Menghan Xia
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-05-30T04:34:08.016Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 64757cbebb0e9dd7764b47d1
paper_authors.name: Haoxin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-19T09:24:20.110Z
paper_authors._id: 6305cb2f435ec751b723d80a
paper_authors.avatarUrl: /avatars/5e8519d9814038726bf98f4104f74568.svg
paper_authors.fullname: Haoxin Chen
paper_authors.type: user
paper_authors.user: paulai
paper_authors._id: 64757cbebb0e9dd7764b47d2
paper_authors.name: Longyue Wang
paper_authors._id: 64757cbebb0e9dd7764b47d3
paper_authors.name: Yong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T09:10:38.643Z
paper_authors._id: 6311b3f3fdb55de45d1fd2bd
paper_authors.avatarUrl: /avatars/855597bd5814d2aa7818da7e70ea2694.svg
paper_authors.fullname: F
paper_authors.type: user
paper_authors.user: norris
paper_authors._id: 64757cbebb0e9dd7764b47d4
paper_authors.name: Xintao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T08:21:27.939Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 64757cbebb0e9dd7764b47d5
paper_authors.name: Ying Shan
paper_authors._id: 64757cbebb0e9dd7764b47d6
paper_authors.name: Yujiu Yang
paper_title: TaleCrafter: Interactive Story Visualization with Multiple Characters
paper_summary: Accurate Story visualization requires several necessary elements, such as
  identity consistency across frames, the alignment between plain text and visual
  content, and a reasonable layout of objects in images. Most previous works
  endeavor to meet these requirements by fitting a text-to-image (T2I) model on a
  set of videos in the same style and with the same characters, e.g., the
  FlintstonesSV dataset. However, the learned T2I models typically struggle to
  adapt to new characters, scenes, and styles, and often lack the flexibility to
  revise the layout of the synthesized images. This paper proposes a system for
  generic interactive story visualization, capable of handling multiple novel
  characters and supporting the editing of layout and local structure. It is
  developed by leveraging the prior knowledge of large language and T2I models,
  trained on massive corpora. The system comprises four interconnected
  components: story-to-prompt generation (S2P), text-to-layout generation (T2L),
  controllable text-to-image generation (C-T2I), and image-to-video animation
  (I2V). First, the S2P module converts concise story information into detailed
  prompts required for subsequent stages. Next, T2L generates diverse and
  reasonable layouts based on the prompts, offering users the ability to adjust
  and refine the layout to their preference. The core component, C-T2I, enables
  the creation of images guided by layouts, sketches, and actor-specific
  identifiers to maintain consistency and detail across visualizations. Finally,
  I2V enriches the visualization process by animating the generated images.
  Extensive experiments and a user study are conducted to validate the
  effectiveness and flexibility of interactive editing of the proposed system.
paper_ai_keywords: story-to-prompt generation (S2P) text-to-layout generation (T2L) controllable text-to-image generation (C-T2I) image-to-video animation (I2V) text-to-image (T2I) model pre alignment identity consistency layout of objects story visualization large language model T2I models actor-specific identifiers