title: FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video
    Synthesis
summary: Diffusion models have transformed the image-to-image (I2I) synthesis and are
  now permeating into videos. However, the advancement of video-to-video (V2V)
  synthesis has been hampered by the challenge of maintaining temporal
  consistency across video frames. This paper proposes a consistent V2V synthesis
  framework by jointly leveraging spatial conditions and temporal optical flow
  clues within the source video. Contrary to prior methods that strictly adhere
  to optical flow, our approach harnesses its benefits while handling the
  imperfection in flow estimation. We encode the optical flow via warping from
  the first frame and serve it as a supplementary reference in the diffusion
  model. This enables our model for video synthesis by editing the first frame
  with any prevalent I2I models and then propagating edits to successive frames.
  Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility:
  FlowVid works seamlessly with existing I2I models, facilitating various
  modifications, including stylization, object swaps, and local edits. (2)
  Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution
  takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF,
  Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our
  FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender
  (10.2%), and TokenFlow (40.4%).
paper_authors._id: 659234bd11f68f12ea863474
paper_authors.name: Feng Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-02T09:27:51.577Z
paper_authors._id: 63631be58f43a912fc7595d5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667440137801-63631be58f43a912fc7595d5.jpeg
paper_authors.fullname: Jeff Liang
paper_authors.type: user
paper_authors.user: JeffLiang
paper_authors._id: 659234bd11f68f12ea863475
paper_authors.name: Bichen Wu
paper_authors._id: 659234bd11f68f12ea863476
paper_authors.name: Jialiang Wang
paper_authors._id: 659234bd11f68f12ea863477
paper_authors.name: Licheng Yu
paper_authors._id: 659234bd11f68f12ea863478
paper_authors.name: Kunpeng Li
paper_authors._id: 659234bd11f68f12ea863479
paper_authors.name: Yinan Zhao
paper_authors._id: 659234bd11f68f12ea86347a
paper_authors.name: Ishan Misra
paper_authors._id: 659234bd11f68f12ea86347b
paper_authors.name: Jia-Bin Huang
paper_authors._id: 659234bd11f68f12ea86347c
paper_authors.name: Peizhao Zhang
paper_authors._id: 659234bd11f68f12ea86347d
paper_authors.name: Peter Vajda
paper_authors._id: 659234bd11f68f12ea86347e
paper_authors.name: Diana Marculescu
paper_title: FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video
    Synthesis
paper_summary: Diffusion models have transformed the image-to-image (I2I) synthesis and are
  now permeating into videos. However, the advancement of video-to-video (V2V)
  synthesis has been hampered by the challenge of maintaining temporal
  consistency across video frames. This paper proposes a consistent V2V synthesis
  framework by jointly leveraging spatial conditions and temporal optical flow
  clues within the source video. Contrary to prior methods that strictly adhere
  to optical flow, our approach harnesses its benefits while handling the
  imperfection in flow estimation. We encode the optical flow via warping from
  the first frame and serve it as a supplementary reference in the diffusion
  model. This enables our model for video synthesis by editing the first frame
  with any prevalent I2I models and then propagating edits to successive frames.
  Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility:
  FlowVid works seamlessly with existing I2I models, facilitating various
  modifications, including stylization, object swaps, and local edits. (2)
  Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution
  takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF,
  Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our
  FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender
  (10.2%), and TokenFlow (40.4%).
paper_ai_summary: A proposed consistent V2V synthesis framework uses diffusion models combined with encoded optical flow to propagate edits across video frames, offering flexibility, efficiency, and high-quality results.
paper_ai_keywords: diffusion models video-to-video (V2V) synthesis temporal optical flow spatial conditions warping FlowVid