title: Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
summary: Large text-to-image diffusion models have exhibited impressive proficiency in
  generating high-quality images. However, when applying these models to video
  domain, ensuring temporal consistency across video frames remains a formidable
  challenge. This paper proposes a novel zero-shot text-guided video-to-video
  translation framework to adapt image models to videos. The framework includes
  two parts: key frame translation and full video translation. The first part
  uses an adapted diffusion model to generate key frames, with hierarchical
  cross-frame constraints applied to enforce coherence in shapes, textures and
  colors. The second part propagates the key frames to other frames with
  temporal-aware patch matching and frame blending. Our framework achieves global
  style and local texture temporal consistency at a low cost (without re-training
  or optimization). The adaptation is compatible with existing image diffusion
  techniques, allowing our framework to take advantage of them, such as
  customizing a specific subject with LoRA, and introducing extra spatial
  guidance with ControlNet. Extensive experimental results demonstrate the
  effectiveness of our proposed framework over existing methods in rendering
  high-quality and temporally-coherent videos.
paper_authors._id: 6489262f50aa32474db581de
paper_authors.name: Shuai Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T08:12:52.839Z
paper_authors._id: 62a54d0410334c1d024e2f59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
paper_authors.fullname: Shuai Yang
paper_authors.type: user
paper_authors.user: PKUWilliamYang
paper_authors._id: 6489262f50aa32474db581df
paper_authors.name: Yifan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T12:52:49.347Z
paper_authors._id: 64a63f9449b08110f761cd73
paper_authors.avatarUrl: /avatars/61860202fc818b105ef24e74dd4f7d3c.svg
paper_authors.fullname: Yifan Zhou
paper_authors.type: user
paper_authors.user: SingleZombie
paper_authors._id: 6489262f50aa32474db581e0
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-14T08:14:22.224Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 6489262f50aa32474db581e1
paper_authors.name: Chen Change Loy
paper_title: Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
paper_summary: Large text-to-image diffusion models have exhibited impressive proficiency in
  generating high-quality images. However, when applying these models to video
  domain, ensuring temporal consistency across video frames remains a formidable
  challenge. This paper proposes a novel zero-shot text-guided video-to-video
  translation framework to adapt image models to videos. The framework includes
  two parts: key frame translation and full video translation. The first part
  uses an adapted diffusion model to generate key frames, with hierarchical
  cross-frame constraints applied to enforce coherence in shapes, textures and
  colors. The second part propagates the key frames to other frames with
  temporal-aware patch matching and frame blending. Our framework achieves global
  style and local texture temporal consistency at a low cost (without re-training
  or optimization). The adaptation is compatible with existing image diffusion
  techniques, allowing our framework to take advantage of them, such as
  customizing a specific subject with LoRA, and introducing extra spatial
  guidance with ControlNet. Extensive experimental results demonstrate the
  effectiveness of our proposed framework over existing methods in rendering
  high-quality and temporally-coherent videos.
paper_ai_summary: A novel framework adapts image diffusion models for video by generating key frames with hierarchical constraints and propagating them using patch matching and blending, achieving high-quality and temporally-coherent videos.
paper_ai_keywords: diffusion models text-guided video-to-video translation key frame translation hierarchical cross-frame constraints temporal-aware patch matching frame blending global style consistency local texture consistency LoRA ControlNet