title: VidToMe: Video Token Merging for Zero-Shot Video Editing
summary: Diffusion models have made significant advances in generating high-quality
  images, but their application to video generation has remained challenging due
  to the complexity of temporal motion. Zero-shot video editing offers a solution
  by utilizing pre-trained image diffusion models to translate source videos into
  new ones. Nevertheless, existing methods struggle to maintain strict temporal
  consistency and efficient memory consumption. In this work, we propose a novel
  approach to enhance temporal consistency in generated videos by merging
  self-attention tokens across frames. By aligning and compressing temporally
  redundant tokens across frames, our method improves temporal coherence and
  reduces memory consumption in self-attention computations. The merging strategy
  matches and aligns tokens according to the temporal correspondence between
  frames, facilitating natural temporal consistency in generated video frames. To
  manage the complexity of video processing, we divide videos into chunks and
  develop intra-chunk local token merging and inter-chunk global token merging,
  ensuring both short-term video continuity and long-term content consistency.
  Our video editing approach seamlessly extends the advancements in image editing
  to video editing, rendering favorable results in temporal consistency over
  state-of-the-art methods.
paper_authors._id: 6581287a23a7aac397d674eb
paper_authors.name: Xirui Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:08:05.579Z
paper_authors._id: 6451364c4b84eee0deea68fa
paper_authors.avatarUrl: /avatars/cc5c52f9b30f43b7faa47d0d5848a492.svg
paper_authors.fullname: Xirui Li
paper_authors.type: user
paper_authors.user: lixirui142
paper_authors._id: 6581287a23a7aac397d674ec
paper_authors.name: Chao Ma
paper_authors._id: 6581287a23a7aac397d674ed
paper_authors.name: Xiaokang Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:08:56.434Z
paper_authors._id: 64eaba6b1d8b651a9aed1203
paper_authors.avatarUrl: /avatars/a0f26095c262ca1a204c27ff4bfb188c.svg
paper_authors.fullname: yangxiaokang
paper_authors.type: user
paper_authors.user: zicheqingluo
paper_authors._id: 6581287a23a7aac397d674ee
paper_authors.name: Ming-Hsuan Yang
paper_title: VidToMe: Video Token Merging for Zero-Shot Video Editing
paper_summary: Diffusion models have made significant advances in generating high-quality
  images, but their application to video generation has remained challenging due
  to the complexity of temporal motion. Zero-shot video editing offers a solution
  by utilizing pre-trained image diffusion models to translate source videos into
  new ones. Nevertheless, existing methods struggle to maintain strict temporal
  consistency and efficient memory consumption. In this work, we propose a novel
  approach to enhance temporal consistency in generated videos by merging
  self-attention tokens across frames. By aligning and compressing temporally
  redundant tokens across frames, our method improves temporal coherence and
  reduces memory consumption in self-attention computations. The merging strategy
  matches and aligns tokens according to the temporal correspondence between
  frames, facilitating natural temporal consistency in generated video frames. To
  manage the complexity of video processing, we divide videos into chunks and
  develop intra-chunk local token merging and inter-chunk global token merging,
  ensuring both short-term video continuity and long-term content consistency.
  Our video editing approach seamlessly extends the advancements in image editing
  to video editing, rendering favorable results in temporal consistency over
  state-of-the-art methods.
paper_ai_summary: A novel method improves temporal consistency and reduces memory consumption in video generated by diffusion models through token merging across frames.
paper_ai_keywords: diffusion models zero-shot video editing self-attention temporal consistency token merging temporal coherence intra-chunk local token merging inter-chunk global token merging short-term video continuity long-term content consistency