title: SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes
summary: Existing methods for the 4D reconstruction of general, non-rigidly deforming
  objects focus on novel-view synthesis and neglect correspondences. However,
  time consistency enables advanced downstream tasks like 3D editing, motion
  analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a
  general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method
  takes multi-view RGB videos and background images from static cameras with
  known camera parameters as input. It then reconstructs the deformations of an
  estimated canonical model of the geometry and appearance in an online fashion.
  Since this canonical model is time-invariant, we obtain correspondences even
  for long-term, long-range motions. We employ neural scene representations to
  parametrize the components of our method. Like prior dynamic-NeRF methods, we
  use a backwards deformation model. We find non-trivial adaptations of this
  model necessary to handle larger motions: We decompose the deformations into a
  strongly regularized coarse component and a weakly regularized fine component,
  where the coarse component also extends the deformation field into the space
  surrounding the object, which enables tracking over time. We show
  experimentally that, unlike prior work that only handles small motion, our
  method enables the reconstruction of studio-scale motions.
paper_authors._id: 64ddc9db103862a42dfe140d
paper_authors.name: Edith Tretschk
paper_authors._id: 64ddc9db103862a42dfe140e
paper_authors.name: Vladislav Golyanik
paper_authors._id: 64ddc9db103862a42dfe140f
paper_authors.name: Michael Zollhoefer
paper_authors._id: 64ddc9db103862a42dfe1410
paper_authors.name: Aljaz Bozic
paper_authors._id: 64ddc9db103862a42dfe1411
paper_authors.name: Christoph Lassner
paper_authors._id: 64ddc9db103862a42dfe1412
paper_authors.name: Christian Theobalt
paper_title: SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes
paper_summary: Existing methods for the 4D reconstruction of general, non-rigidly deforming
  objects focus on novel-view synthesis and neglect correspondences. However,
  time consistency enables advanced downstream tasks like 3D editing, motion
  analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a
  general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method
  takes multi-view RGB videos and background images from static cameras with
  known camera parameters as input. It then reconstructs the deformations of an
  estimated canonical model of the geometry and appearance in an online fashion.
  Since this canonical model is time-invariant, we obtain correspondences even
  for long-term, long-range motions. We employ neural scene representations to
  parametrize the components of our method. Like prior dynamic-NeRF methods, we
  use a backwards deformation model. We find non-trivial adaptations of this
  model necessary to handle larger motions: We decompose the deformations into a
  strongly regularized coarse component and a weakly regularized fine component,
  where the coarse component also extends the deformation field into the space
  surrounding the object, which enables tracking over time. We show
  experimentally that, unlike prior work that only handles small motion, our
  method enables the reconstruction of studio-scale motions.
paper_ai_summary: SceNeRFlow reconstructs non-rigid time-consistent scenes using dynamic-NeRF with a decomposed deformation model to handle larger motions.
paper_ai_keywords: dynamic-NeRF neural scene representations backwards deformation model time consistency canonical model coarse component fine component studio-scale motions