title: AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
summary: Large-scale text-to-video (T2V) diffusion models have great progress in
  recent years in terms of visual quality, motion and temporal consistency.
  However, the generation process is still a black box, where all attributes
  (e.g., appearance, motion) are learned and generated jointly without precise
  control ability other than rough text descriptions. Inspired by image animation
  which decouples the video as one specific appearance with the corresponding
  motion, we propose AnimateZero to unveil the pre-trained text-to-video
  diffusion model, i.e., AnimateDiff, and provide more precise appearance and
  motion control abilities for it. For appearance control, we borrow intermediate
  latents and their features from the text-to-image (T2I) generation for ensuring
  the generated first frame is equal to the given generated image. For temporal
  control, we replace the global temporal attention of the original T2V model
  with our proposed positional-corrected window attention to ensure other frames
  align with the first frame well. Empowered by the proposed methods, AnimateZero
  can successfully control the generating progress without further training. As a
  zero-shot image animator for given images, AnimateZero also enables multiple
  new applications, including interactive video generation and real image
  animation. The detailed experiments demonstrate the effectiveness of the
  proposed method in both T2V and related applications.
paper_authors._id: 6572907e990936e2e9105ec2
paper_authors.name: Jiwen Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:09:54.581Z
paper_authors._id: 64105a6d14215c0775dfdd14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64105a6d14215c0775dfdd14/-VX-cUYOLjHIg7QnWhRGG.jpeg
paper_authors.fullname: Jiwen Yu
paper_authors.type: user
paper_authors.user: VictorYuki
paper_authors._id: 6572907e990936e2e9105ec3
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:10:03.681Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 6572907e990936e2e9105ec4
paper_authors.name: Chenyang Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:10:11.359Z
paper_authors._id: 642157a201c62c1e41fc526a
paper_authors.avatarUrl: /avatars/73ae831a73947a04468b1e87d17da24d.svg
paper_authors.fullname: chenyangqi
paper_authors.type: user
paper_authors.user: chenyangqi
paper_authors._id: 6572907e990936e2e9105ec5
paper_authors.name: Yong Zhang
paper_authors._id: 6572907e990936e2e9105ec6
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:10:59.060Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 6572907e990936e2e9105ec7
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:11:16.309Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 6572907e990936e2e9105ec8
paper_authors.name: Jian Zhang
paper_title: AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
paper_summary: Large-scale text-to-video (T2V) diffusion models have great progress in
  recent years in terms of visual quality, motion and temporal consistency.
  However, the generation process is still a black box, where all attributes
  (e.g., appearance, motion) are learned and generated jointly without precise
  control ability other than rough text descriptions. Inspired by image animation
  which decouples the video as one specific appearance with the corresponding
  motion, we propose AnimateZero to unveil the pre-trained text-to-video
  diffusion model, i.e., AnimateDiff, and provide more precise appearance and
  motion control abilities for it. For appearance control, we borrow intermediate
  latents and their features from the text-to-image (T2I) generation for ensuring
  the generated first frame is equal to the given generated image. For temporal
  control, we replace the global temporal attention of the original T2V model
  with our proposed positional-corrected window attention to ensure other frames
  align with the first frame well. Empowered by the proposed methods, AnimateZero
  can successfully control the generating progress without further training. As a
  zero-shot image animator for given images, AnimateZero also enables multiple
  new applications, including interactive video generation and real image
  animation. The detailed experiments demonstrate the effectiveness of the
  proposed method in both T2V and related applications.
paper_ai_summary: AnimateZero enhances pre-trained text-to-video diffusion models with precise appearance and motion control capabilities, enabling finer control over the generation process and supporting new applications like interactive video generation.
paper_ai_keywords: text-to-video diffusion models appearance control intermediate latents text-to-image generation temporal control global temporal attention positional-corrected window attention zero-shot image animator interactive video generation real image animation