title: Generative Inbetweening: Adapting Image-to-Video Models for Keyframe
    Interpolation
summary: We present a method for generating video sequences with coherent motion
  between a pair of input key frames. We adapt a pretrained large-scale
  image-to-video diffusion model (originally trained to generate videos moving
  forward in time from a single input image) for key frame interpolation, i.e.,
  to produce a video in between two input frames. We accomplish this adaptation
  through a lightweight fine-tuning technique that produces a version of the
  model that instead predicts videos moving backwards in time from a single input
  image. This model (along with the original forward-moving model) is
  subsequently used in a dual-directional diffusion sampling process that
  combines the overlapping model estimates starting from each of the two
  keyframes. Our experiments show that our method outperforms both existing
  diffusion-based methods and traditional frame interpolation techniques.
paper_authors._id: 66ce96ed6116c7f53e64c579
paper_authors.name: Xiaojuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:26:25.939Z
paper_authors._id: 643ee9a6022a0350efd745d1
paper_authors.avatarUrl: /avatars/c8c17689ed88b1966d2a36361e3f83cc.svg
paper_authors.fullname: Xiaojuan Wang
paper_authors.type: user
paper_authors.user: xiaojwan
paper_authors._id: 66ce96ed6116c7f53e64c57a
paper_authors.name: Boyang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:26:33.427Z
paper_authors._id: 6445f1105691ca69b0df48fc
paper_authors.avatarUrl: /avatars/0e2fc769a0c044d0dd9a9c0debf49782.svg
paper_authors.fullname: Boyang Zhou
paper_authors.type: user
paper_authors.user: BobbyZ963
paper_authors._id: 66ce96ed6116c7f53e64c57b
paper_authors.name: Brian Curless
paper_authors._id: 66ce96ed6116c7f53e64c57c
paper_authors.name: Ira Kemelmacher-Shlizerman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:19:03.265Z
paper_authors._id: 66d20e411ba71ac4c0488132
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/kkqDAFT15lx4wrJVXiFGJ.png
paper_authors.fullname: Ira Kemelmacher-Shlizerman
paper_authors.type: user
paper_authors.user: kemelmi
paper_authors._id: 66ce96ed6116c7f53e64c57d
paper_authors.name: Aleksander Holynski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:27:26.657Z
paper_authors._id: 62f6e2792e53c2efd33faa92
paper_authors.avatarUrl: /avatars/85d94b4022577747b8d2d10a82c2f3c7.svg
paper_authors.fullname: Aleksander Holynski
paper_authors.type: user
paper_authors.user: holynski
paper_authors._id: 66ce96ed6116c7f53e64c57e
paper_authors.name: Steven M. Seitz
paper_title: Generative Inbetweening: Adapting Image-to-Video Models for Keyframe
    Interpolation
paper_summary: We present a method for generating video sequences with coherent motion
  between a pair of input key frames. We adapt a pretrained large-scale
  image-to-video diffusion model (originally trained to generate videos moving
  forward in time from a single input image) for key frame interpolation, i.e.,
  to produce a video in between two input frames. We accomplish this adaptation
  through a lightweight fine-tuning technique that produces a version of the
  model that instead predicts videos moving backwards in time from a single input
  image. This model (along with the original forward-moving model) is
  subsequently used in a dual-directional diffusion sampling process that
  combines the overlapping model estimates starting from each of the two
  keyframes. Our experiments show that our method outperforms both existing
  diffusion-based methods and traditional frame interpolation techniques.
paper_ai_summary: A method for generating coherent video sequences between key frames using a dual-directional diffusion process outperforms existing techniques.
paper_ai_keywords: diffusion model key frame interpolation fine-tuning dual-directional diffusion sampling video generation