title: ReCapture: Generative Video Camera Controls for User-Provided Videos
    using Masked Video Fine-Tuning
summary: Recently, breakthroughs in video modeling have allowed for controllable
  camera trajectories in generated videos. However, these methods cannot be
  directly applied to user-provided videos that are not generated by a video
  model. In this paper, we present ReCapture, a method for generating new videos
  with novel camera trajectories from a single user-provided video. Our method
  allows us to re-generate the reference video, with all its existing scene
  motion, from vastly different angles and with cinematic camera motion. Notably,
  using our method we can also plausibly hallucinate parts of the scene that were
  not observable in the reference video. Our method works by (1) generating a
  noisy anchor video with a new camera trajectory using multiview diffusion
  models or depth-based point cloud rendering and then (2) regenerating the
  anchor video into a clean and temporally consistent reangled video using our
  proposed masked video fine-tuning technique.
paper_authors._id: 672d74072dbda1cf55a7bd44
paper_authors.name: David Junhao Zhang
paper_authors._id: 672d74072dbda1cf55a7bd45
paper_authors.name: Roni Paiss
paper_authors._id: 672d74072dbda1cf55a7bd46
paper_authors.name: Shiran Zada
paper_authors._id: 672d74072dbda1cf55a7bd47
paper_authors.name: Nikhil Karnad
paper_authors._id: 672d74072dbda1cf55a7bd48
paper_authors.name: David E. Jacobs
paper_authors._id: 672d74072dbda1cf55a7bd49
paper_authors.name: Yael Pritch
paper_authors._id: 672d74072dbda1cf55a7bd4a
paper_authors.name: Inbar Mosseri
paper_authors._id: 672d74072dbda1cf55a7bd4b
paper_authors.name: Mike Zheng Shou
paper_authors._id: 672d74072dbda1cf55a7bd4c
paper_authors.name: Neal Wadhwa
paper_authors._id: 672d74072dbda1cf55a7bd4d
paper_authors.name: Nataniel Ruiz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:31:56.991Z
paper_authors._id: 6318c6e6cb116eab31d70a16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
paper_authors.fullname: Nataniel
paper_authors.type: user
paper_authors.user: flavoredquark
paper_title: ReCapture: Generative Video Camera Controls for User-Provided Videos
    using Masked Video Fine-Tuning
paper_summary: Recently, breakthroughs in video modeling have allowed for controllable
  camera trajectories in generated videos. However, these methods cannot be
  directly applied to user-provided videos that are not generated by a video
  model. In this paper, we present ReCapture, a method for generating new videos
  with novel camera trajectories from a single user-provided video. Our method
  allows us to re-generate the reference video, with all its existing scene
  motion, from vastly different angles and with cinematic camera motion. Notably,
  using our method we can also plausibly hallucinate parts of the scene that were
  not observable in the reference video. Our method works by (1) generating a
  noisy anchor video with a new camera trajectory using multiview diffusion
  models or depth-based point cloud rendering and then (2) regenerating the
  anchor video into a clean and temporally consistent reangled video using our
  proposed masked video fine-tuning technique.
paper_ai_keywords: multiview diffusion models depth-based point cloud rendering masked video fine-tuning