Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
Generating videos for visual storytelling can be a tedious and complex
process that typically requires either live-action filming or graphics
animation rendering. To bypass these challenges, our key idea is to utilize the
abundance of existing video clips and synthesize a coherent storytelling video
by customizing their appearances. We achieve this by developing a framework
comprised of two functional modules: (i) Motion Structure Retrieval, which
provides video candidates with desired scene or motion context described by
query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates
plot-aligned videos under the guidance of motion structure and text prompts.
For the first module, we leverage an off-the-shelf video retrieval system and
extract video depths as motion structure. For the second module, we propose a
controllable video generation model that offers flexible controls over
structure and characters. The videos are synthesized by following the
structural guidance and appearance instruction. To ensure visual consistency
across clips, we propose an effective concept personalization approach, which
allows the specification of the desired character identities through text
prompts. Extensive experiments demonstrate that our approach exhibits
significant advantages over various existing baselines.
64b0afbc1ab21d61b6602ae7
Yingqing He
64b0afbc1ab21d61b6602ae8
Menghan Xia
admin_assigned
2023-07-17T11:16:46.768Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
64b0afbc1ab21d61b6602ae9
Haoxin Chen
claimed_verified
2024-01-19T09:24:10.052Z
6305cb2f435ec751b723d80a
/avatars/5e8519d9814038726bf98f4104f74568.svg
Haoxin Chen
user
paulai
64b0afbc1ab21d61b6602aea
Xiaodong Cun
admin_assigned
2023-07-17T11:17:45.179Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
64b0afbc1ab21d61b6602aeb
Yuan Gong
64b0afbc1ab21d61b6602aec
Jinbo Xing
claimed_verified
2025-05-14T10:04:16.568Z
64770e86d7cf39f2e937ae9a
https://cdn-avatars.huggingface.co/v1/production/uploads/64770e86d7cf39f2e937ae9a/pLqGg2z1KzQxCGpMwds-9.jpeg
Jinbo Xing
user
Doubiiu
64b0afbc1ab21d61b6602aed
Yong Zhang
64b0afbc1ab21d61b6602aee
Xintao Wang
admin_assigned
2023-07-17T11:19:16.935Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
64b0afbc1ab21d61b6602aef
Chao Weng
64b0afbc1ab21d61b6602af0
Ying Shan
64b0afbc1ab21d61b6602af1
Qifeng Chen
admin_assigned
2023-07-17T11:20:15.211Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation
Generating videos for visual storytelling can be a tedious and complex
process that typically requires either live-action filming or graphics
animation rendering. To bypass these challenges, our key idea is to utilize the
abundance of existing video clips and synthesize a coherent storytelling video
by customizing their appearances. We achieve this by developing a framework
comprised of two functional modules: (i) Motion Structure Retrieval, which
provides video candidates with desired scene or motion context described by
query texts, and (ii) Structure-Guided Text-to-Video Synthesis, which generates
plot-aligned videos under the guidance of motion structure and text prompts.
For the first module, we leverage an off-the-shelf video retrieval system and
extract video depths as motion structure. For the second module, we propose a
controllable video generation model that offers flexible controls over
structure and characters. The videos are synthesized by following the
structural guidance and appearance instruction. To ensure visual consistency
across clips, we propose an effective concept personalization approach, which
allows the specification of the desired character identities through text
prompts. Extensive experiments demonstrate that our approach exhibits
significant advantages over various existing baselines.
A framework utilizes existing video clips to generate coherent storytelling videos by combining motion structure retrieval and text-guided video synthesis, offering flexibility and consistency.
motion structure retrieval text-to-video synthesis video retrieval system video depth concept personalization character identity text prompts video synthesis