title: FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
summary: The remarkable efficacy of text-to-image diffusion models has motivated
  extensive exploration of their potential application in video domains.
  Zero-shot methods seek to extend image diffusion models to videos without
  necessitating model training. Recent methods mainly focus on incorporating
  inter-frame correspondence into attention mechanisms. However, the soft
  constraint imposed on determining where to attend to valid features can
  sometimes be insufficient, resulting in temporal inconsistency. In this paper,
  we introduce FRESCO, intra-frame correspondence alongside inter-frame
  correspondence to establish a more robust spatial-temporal constraint. This
  enhancement ensures a more consistent transformation of semantically similar
  content across frames. Beyond mere attention guidance, our approach involves an
  explicit update of features to achieve high spatial-temporal consistency with
  the input video, significantly improving the visual coherence of the resulting
  translated videos. Extensive experiments demonstrate the effectiveness of our
  proposed framework in producing high-quality, coherent videos, marking a
  notable improvement over existing zero-shot methods.
paper_authors._id: 65fa679becb85d8ad7e030c2
paper_authors.name: Shuai Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T13:15:19.551Z
paper_authors._id: 62a54d0410334c1d024e2f59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664764278226-62a54d0410334c1d024e2f59.jpeg
paper_authors.fullname: Shuai Yang
paper_authors.type: user
paper_authors.user: PKUWilliamYang
paper_authors._id: 65fa679becb85d8ad7e030c3
paper_authors.name: Yifan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T12:52:46.241Z
paper_authors._id: 64a63f9449b08110f761cd73
paper_authors.avatarUrl: /avatars/61860202fc818b105ef24e74dd4f7d3c.svg
paper_authors.fullname: Yifan Zhou
paper_authors.type: user
paper_authors.user: SingleZombie
paper_authors._id: 65fa679becb85d8ad7e030c4
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T13:15:59.554Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 65fa679becb85d8ad7e030c5
paper_authors.name: Chen Change Loy
paper_title: FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
paper_summary: The remarkable efficacy of text-to-image diffusion models has motivated
  extensive exploration of their potential application in video domains.
  Zero-shot methods seek to extend image diffusion models to videos without
  necessitating model training. Recent methods mainly focus on incorporating
  inter-frame correspondence into attention mechanisms. However, the soft
  constraint imposed on determining where to attend to valid features can
  sometimes be insufficient, resulting in temporal inconsistency. In this paper,
  we introduce FRESCO, intra-frame correspondence alongside inter-frame
  correspondence to establish a more robust spatial-temporal constraint. This
  enhancement ensures a more consistent transformation of semantically similar
  content across frames. Beyond mere attention guidance, our approach involves an
  explicit update of features to achieve high spatial-temporal consistency with
  the input video, significantly improving the visual coherence of the resulting
  translated videos. Extensive experiments demonstrate the effectiveness of our
  proposed framework in producing high-quality, coherent videos, marking a
  notable improvement over existing zero-shot methods.
paper_ai_keywords: diffusion models text-to-image video domains zero-shot methods inter-frame correspondence attention mechanisms intra-frame correspondence spatial-temporal constraint semantically similar visual coherence high-quality coherent videos