title: Controlling Space and Time with Diffusion Models
summary: We present 4DiM, a cascaded diffusion model for 4D novel view synthesis
  (NVS), conditioned on one or more images of a general scene, and a set of
  camera poses and timestamps. To overcome challenges due to limited availability
  of 4D training data, we advocate joint training on 3D (with camera pose), 4D
  (pose+time) and video (time but no pose) data and propose a new architecture
  that enables the same. We further advocate the calibration of SfM posed data
  using monocular metric depth estimators for metric scale camera control. For
  model evaluation, we introduce new metrics to enrich and overcome shortcomings
  of current evaluation schemes, demonstrating state-of-the-art results in both
  fidelity and pose control compared to existing diffusion models for 3D NVS,
  while at the same time adding the ability to handle temporal dynamics. 4DiM is
  also used for improved panorama stitching, pose-conditioned video to video
  translation, and several other tasks. For an overview see
  https://4d-diffusion.github.io
paper_authors._id: 668f4bdd27e286af06887a7e
paper_authors.name: Daniel Watson
paper_authors._id: 668f4bdd27e286af06887a7f
paper_authors.name: Saurabh Saxena
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-11T10:42:35.081Z
paper_authors._id: 6480af48de559d48dbb20d95
paper_authors.avatarUrl: /avatars/69f1e19921ba3ecdcd40bf55a6196aca.svg
paper_authors.fullname: Saurabh Saxena
paper_authors.type: user
paper_authors.user: saurabhsaxena
paper_authors._id: 668f4bdd27e286af06887a80
paper_authors.name: Lala Li
paper_authors._id: 668f4bdd27e286af06887a81
paper_authors.name: Andrea Tagliasacchi
paper_authors._id: 668f4bdd27e286af06887a82
paper_authors.name: David J. Fleet
paper_title: Controlling Space and Time with Diffusion Models
paper_summary: We present 4DiM, a cascaded diffusion model for 4D novel view synthesis
  (NVS), conditioned on one or more images of a general scene, and a set of
  camera poses and timestamps. To overcome challenges due to limited availability
  of 4D training data, we advocate joint training on 3D (with camera pose), 4D
  (pose+time) and video (time but no pose) data and propose a new architecture
  that enables the same. We further advocate the calibration of SfM posed data
  using monocular metric depth estimators for metric scale camera control. For
  model evaluation, we introduce new metrics to enrich and overcome shortcomings
  of current evaluation schemes, demonstrating state-of-the-art results in both
  fidelity and pose control compared to existing diffusion models for 3D NVS,
  while at the same time adding the ability to handle temporal dynamics. 4DiM is
  also used for improved panorama stitching, pose-conditioned video to video
  translation, and several other tasks. For an overview see
  https://4d-diffusion.github.io
paper_ai_keywords: cascaded diffusion model 4D novel view synthesis camera poses timestamps 3D NVS joint training monocular metric depth estimators metric scale camera control panorama stitching pose-conditioned video to video translation