title: Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
summary: Generating videos for visual storytelling can be a tedious and complex
  process that typically requires either live-action filming or graphics
  animation rendering. To bypass these challenges, our key idea is to utilize the
  abundance of existing video clips and synthesize a coherent storytelling video
  by customizing their appearances. We achieve this by developing a framework
  comprised of two functional modules: (i) Motion Structure Retrieval, which
  provides video candidates with desired scene or motion context described by
  query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates
  plot-aligned videos under the guidance of motion structure and text prompts.
  For the first module, we leverage an off-the-shelf video retrieval system and
  extract video depths as motion structure. For the second module, we propose a
  controllable video generation model that offers flexible controls over
  structure and characters. The videos are synthesized by following the
  structural guidance and appearance instruction. To ensure visual consistency
  across clips, we propose an effective concept personalization approach, which
  allows the specification of the desired character identities through text
  prompts. Extensive experiments demonstrate that our approach exhibits
  significant advantages over various existing baselines.
paper_authors._id: 64b0afbc1ab21d61b6602ae7
paper_authors.name: Yingqing He
paper_authors._id: 64b0afbc1ab21d61b6602ae8
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:16:46.768Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 64b0afbc1ab21d61b6602ae9
paper_authors.name: Haoxin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-19T09:24:10.052Z
paper_authors._id: 6305cb2f435ec751b723d80a
paper_authors.avatarUrl: /avatars/5e8519d9814038726bf98f4104f74568.svg
paper_authors.fullname: Haoxin Chen
paper_authors.type: user
paper_authors.user: paulai
paper_authors._id: 64b0afbc1ab21d61b6602aea
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:17:45.179Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 64b0afbc1ab21d61b6602aeb
paper_authors.name: Yuan Gong
paper_authors._id: 64b0afbc1ab21d61b6602aec
paper_authors.name: Jinbo Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T10:04:16.568Z
paper_authors._id: 64770e86d7cf39f2e937ae9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64770e86d7cf39f2e937ae9a/pLqGg2z1KzQxCGpMwds-9.jpeg
paper_authors.fullname: Jinbo Xing
paper_authors.type: user
paper_authors.user: Doubiiu
paper_authors._id: 64b0afbc1ab21d61b6602aed
paper_authors.name: Yong Zhang
paper_authors._id: 64b0afbc1ab21d61b6602aee
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:19:16.935Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 64b0afbc1ab21d61b6602aef
paper_authors.name: Chao Weng
paper_authors._id: 64b0afbc1ab21d61b6602af0
paper_authors.name: Ying Shan
paper_authors._id: 64b0afbc1ab21d61b6602af1
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-17T11:20:15.211Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_title: Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
paper_summary: Generating videos for visual storytelling can be a tedious and complex
  process that typically requires either live-action filming or graphics
  animation rendering. To bypass these challenges, our key idea is to utilize the
  abundance of existing video clips and synthesize a coherent storytelling video
  by customizing their appearances. We achieve this by developing a framework
  comprised of two functional modules: (i) Motion Structure Retrieval, which
  provides video candidates with desired scene or motion context described by
  query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates
  plot-aligned videos under the guidance of motion structure and text prompts.
  For the first module, we leverage an off-the-shelf video retrieval system and
  extract video depths as motion structure. For the second module, we propose a
  controllable video generation model that offers flexible controls over
  structure and characters. The videos are synthesized by following the
  structural guidance and appearance instruction. To ensure visual consistency
  across clips, we propose an effective concept personalization approach, which
  allows the specification of the desired character identities through text
  prompts. Extensive experiments demonstrate that our approach exhibits
  significant advantages over various existing baselines.
paper_ai_summary: A framework utilizes existing video clips to generate coherent storytelling videos by combining motion structure retrieval and text-guided video synthesis, offering flexibility and consistency.
paper_ai_keywords: motion structure retrieval text-to-video synthesis video retrieval system video depth concept personalization character identity text prompts video synthesis