title: Looking Backward: Streaming Video-to-Video Translation with Feature
    Banks
summary: This paper introduces StreamV2V, a diffusion model that achieves real-time
  streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V
  methods using batches to process limited frames, we opt to process frames in a
  streaming fashion, to support unlimited frames. At the heart of StreamV2V lies
  a backward-looking principle that relates the present to the past. This is
  realized by maintaining a feature bank, which archives information from past
  frames. For incoming frames, StreamV2V extends self-attention to include banked
  keys and values and directly fuses similar past features into the output. The
  feature bank is continually updated by merging stored and new features, making
  it compact but informative. StreamV2V stands out for its adaptability and
  efficiency, seamlessly integrating with image diffusion models without
  fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x
  faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative
  metrics and user studies confirm StreamV2V's exceptional ability to maintain
  temporal consistency.
paper_authors._id: 66555ecfc7ff1a0216abd811
paper_authors.name: Feng Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:18:39.485Z
paper_authors._id: 63631be58f43a912fc7595d5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667440137801-63631be58f43a912fc7595d5.jpeg
paper_authors.fullname: Jeff Liang
paper_authors.type: user
paper_authors.user: JeffLiang
paper_authors._id: 66555ecfc7ff1a0216abd812
paper_authors.name: Akio Kodaira
paper_authors._id: 66555ecfc7ff1a0216abd813
paper_authors.name: Chenfeng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:36:11.052Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_authors._id: 66555ecfc7ff1a0216abd814
paper_authors.name: Masayoshi Tomizuka
paper_authors._id: 66555ecfc7ff1a0216abd815
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T09:36:25.609Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 66555ecfc7ff1a0216abd816
paper_authors.name: Diana Marculescu
paper_title: Looking Backward: Streaming Video-to-Video Translation with Feature
    Banks
paper_summary: This paper introduces StreamV2V, a diffusion model that achieves real-time
  streaming video-to-video (V2V) translation with user prompts. Unlike prior V2V
  methods using batches to process limited frames, we opt to process frames in a
  streaming fashion, to support unlimited frames. At the heart of StreamV2V lies
  a backward-looking principle that relates the present to the past. This is
  realized by maintaining a feature bank, which archives information from past
  frames. For incoming frames, StreamV2V extends self-attention to include banked
  keys and values and directly fuses similar past features into the output. The
  feature bank is continually updated by merging stored and new features, making
  it compact but informative. StreamV2V stands out for its adaptability and
  efficiency, seamlessly integrating with image diffusion models without
  fine-tuning. It can run 20 FPS on one A100 GPU, being 15x, 46x, 108x, and 158x
  faster than FlowVid, CoDeF, Rerender, and TokenFlow, respectively. Quantitative
  metrics and user studies confirm StreamV2V's exceptional ability to maintain
  temporal consistency.
paper_ai_keywords: diffusion model streamV2V streaming video-to-video (V2V) backward-looking principle feature bank self-attention temporal consistency