title: Generalizable Implicit Motion Modeling for Video Frame Interpolation
summary: Motion modeling is critical in flow-based Video Frame Interpolation (VFI).
  Existing paradigms either consider linear combinations of bidirectional flows
  or directly predict bilateral flows for given timestamps without exploring
  favorable motion priors, thus lacking the capability of effectively modeling
  spatiotemporal dynamics in real-world videos. To address this limitation, in
  this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel
  and effective approach to motion modeling for VFI. Specifically, to enable GIMM
  as an effective motion modeling paradigm, we design a motion encoding pipeline
  to model spatiotemporal motion latent from bidirectional flows extracted from
  pre-trained flow estimators, effectively representing input-specific motion
  priors. Then, we implicitly predict arbitrary-timestep optical flows within two
  adjacent input frames via an adaptive coordinate-based neural network, with
  spatiotemporal coordinates and motion latent as inputs. Our GIMM can be
  smoothly integrated with existing flow-based VFI works without further
  modifications. We show that GIMM performs better than the current state of the
  art on the VFI benchmarks.
paper_authors._id: 6690905937eadb9c578773e4
paper_authors.name: Zujin Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:00:54.351Z
paper_authors._id: 62f4b57a9437c4d04fb5e6d6
paper_authors.avatarUrl: /avatars/3681e4948fba1317c80bbbf2326a5fcd.svg
paper_authors.fullname: ZUJIN GUO
paper_authors.type: user
paper_authors.user: GSean
paper_authors._id: 6690905937eadb9c578773e5
paper_authors.name: Wei Li
paper_authors._id: 6690905937eadb9c578773e6
paper_authors.name: Chen Change Loy
paper_title: Generalizable Implicit Motion Modeling for Video Frame Interpolation
paper_summary: Motion modeling is critical in flow-based Video Frame Interpolation (VFI).
  Existing paradigms either consider linear combinations of bidirectional flows
  or directly predict bilateral flows for given timestamps without exploring
  favorable motion priors, thus lacking the capability of effectively modeling
  spatiotemporal dynamics in real-world videos. To address this limitation, in
  this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel
  and effective approach to motion modeling for VFI. Specifically, to enable GIMM
  as an effective motion modeling paradigm, we design a motion encoding pipeline
  to model spatiotemporal motion latent from bidirectional flows extracted from
  pre-trained flow estimators, effectively representing input-specific motion
  priors. Then, we implicitly predict arbitrary-timestep optical flows within two
  adjacent input frames via an adaptive coordinate-based neural network, with
  spatiotemporal coordinates and motion latent as inputs. Our GIMM can be
  smoothly integrated with existing flow-based VFI works without further
  modifications. We show that GIMM performs better than the current state of the
  art on the VFI benchmarks.
paper_ai_keywords: Generalizable Implicit Motion Modeling (GIMM) motion encoding pipeline bidirectional flows pre-trained flow estimators motion priors adaptive coordinate-based neural network spatiotemporal coordinates optical flows flow-based Video Frame Interpolation (VFI)