title: Diffusion Priors for Dynamic View Synthesis from Monocular Videos
summary: Dynamic novel view synthesis aims to capture the temporal evolution of visual
  content within videos. Existing methods struggle to distinguishing between
  motion and structure, particularly in scenarios where camera poses are either
  unknown or constrained compared to object motion. Furthermore, with information
  solely from reference images, it is extremely challenging to hallucinate unseen
  regions that are occluded or partially observed in the given videos. To address
  these issues, we first finetune a pretrained RGB-D diffusion model on the video
  frames using a customization technique. Subsequently, we distill the knowledge
  from the finetuned model to a 4D representations encompassing both dynamic and
  static Neural Radiance Fields (NeRF) components. The proposed pipeline achieves
  geometric consistency while preserving the scene identity. We perform thorough
  experiments to evaluate the efficacy of the proposed method qualitatively and
  quantitatively. Our results demonstrate the robustness and utility of our
  approach in challenging cases, further advancing dynamic novel view synthesis.
paper_authors._id: 65a0af0126d1e9df4fe416ce
paper_authors.name: Chaoyang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:16:02.817Z
paper_authors._id: 64c166de11f3f1d23a99c6fd
paper_authors.avatarUrl: /avatars/d4730f70a6f1cb51266862c7a8d54f77.svg
paper_authors.fullname: Chaoyang Wang
paper_authors.type: user
paper_authors.user: cwang9
paper_authors._id: 65a0af0126d1e9df4fe416cf
paper_authors.name: Peiye Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:16:08.804Z
paper_authors._id: 63e473cf9db5da2dc1f4e9ff
paper_authors.avatarUrl: /avatars/6973b4c959ef8bb72c40c611d384a78a.svg
paper_authors.fullname: peiye zhuang
paper_authors.type: user
paper_authors.user: Kelest
paper_authors._id: 65a0af0126d1e9df4fe416d0
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:16:15.192Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 65a0af0126d1e9df4fe416d1
paper_authors.name: Junli Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:16:21.963Z
paper_authors._id: 63f54aa73aa49d8cb97b84bc
paper_authors.avatarUrl: /avatars/c73c5870039611ab9162daad46a1ba20.svg
paper_authors.fullname: junli cao
paper_authors.type: user
paper_authors.user: jlcao2
paper_authors._id: 65a0af0126d1e9df4fe416d2
paper_authors.name: Guocheng Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:16:28.558Z
paper_authors._id: 645fed74335c21d19f3bf76c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645fed74335c21d19f3bf76c/gwVsllRWtSHbg4a1erkdF.jpeg
paper_authors.fullname: Guocheng Qian
paper_authors.type: user
paper_authors.user: guochengqian
paper_authors._id: 65a0af0126d1e9df4fe416d3
paper_authors.name: Hsin-Ying Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:16:34.561Z
paper_authors._id: 631bbd918de8e645af71eed9
paper_authors.avatarUrl: /avatars/1cb94deadc288d07c571c9289085548c.svg
paper_authors.fullname: Hsin-Ying Lee
paper_authors.type: user
paper_authors.user: james371507
paper_authors._id: 65a0af0126d1e9df4fe416d4
paper_authors.name: Sergey Tulyakov
paper_title: Diffusion Priors for Dynamic View Synthesis from Monocular Videos
paper_summary: Dynamic novel view synthesis aims to capture the temporal evolution of visual
  content within videos. Existing methods struggle to distinguishing between
  motion and structure, particularly in scenarios where camera poses are either
  unknown or constrained compared to object motion. Furthermore, with information
  solely from reference images, it is extremely challenging to hallucinate unseen
  regions that are occluded or partially observed in the given videos. To address
  these issues, we first finetune a pretrained RGB-D diffusion model on the video
  frames using a customization technique. Subsequently, we distill the knowledge
  from the finetuned model to a 4D representations encompassing both dynamic and
  static Neural Radiance Fields (NeRF) components. The proposed pipeline achieves
  geometric consistency while preserving the scene identity. We perform thorough
  experiments to evaluate the efficacy of the proposed method qualitatively and
  quantitatively. Our results demonstrate the robustness and utility of our
  approach in challenging cases, further advancing dynamic novel view synthesis.
paper_ai_keywords: dynamic novel view synthesis temporal evolution visual content camera poses object motion hallucinate unseen regions RGB-D diffusion model customization technique distill knowledge 4D representations Neural Radiance Fields (NeRF) dynamic NeRF components static NeRF components geometric consistency scene identity