title: ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image
summary: We introduce a 3D-aware diffusion model, ZeroNVS, for single-image novel view
  synthesis for in-the-wild scenes. While existing methods are designed for
  single objects with masked backgrounds, we propose new techniques to address
  challenges introduced by in-the-wild multi-object scenes with complex
  backgrounds. Specifically, we train a generative prior on a mixture of data
  sources that capture object-centric, indoor, and outdoor scenes. To address
  issues from data mixture such as depth-scale ambiguity, we propose a novel
  camera conditioning parameterization and normalization scheme. Further, we
  observe that Score Distillation Sampling (SDS) tends to truncate the
  distribution of complex backgrounds during distillation of 360-degree scenes,
  and propose "SDS anchoring" to improve the diversity of synthesized novel
  views. Our model sets a new state-of-the-art result in LPIPS on the DTU dataset
  in the zero-shot setting, even outperforming methods specifically trained on
  DTU. We further adapt the challenging Mip-NeRF 360 dataset as a new benchmark
  for single-image novel view synthesis, and demonstrate strong performance in
  this setting. Our code and data are at http://kylesargent.github.io/zeronvs/
paper_authors._id: 653f2185b424289c5f839a0d
paper_authors.name: Kyle Sargent
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:37:25.459Z
paper_authors._id: 6451a5d7938967fd069fd135
paper_authors.avatarUrl: /avatars/56a00b2e6211f678ec040ea379e7387a.svg
paper_authors.fullname: Kyle Sargent
paper_authors.type: user
paper_authors.user: ksahug
paper_authors._id: 653f2185b424289c5f839a0e
paper_authors.name: Zizhang Li
paper_authors._id: 653f2185b424289c5f839a0f
paper_authors.name: Tanmay Shah
paper_authors._id: 653f2185b424289c5f839a10
paper_authors.name: Charles Herrmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:38:03.772Z
paper_authors._id: 64c7f088c864d962eddeaf8c
paper_authors.avatarUrl: /avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
paper_authors.fullname: Charles Herrmann
paper_authors.type: user
paper_authors.user: irwinherrmann
paper_authors._id: 653f2185b424289c5f839a11
paper_authors.name: Hong-Xing Yu
paper_authors._id: 653f2185b424289c5f839a12
paper_authors.name: Yunzhi Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:38:31.287Z
paper_authors._id: 63793aee2cb50dda39d6e73b
paper_authors.avatarUrl: /avatars/1d7eb1a19ce2e18802e9481200eb03f7.svg
paper_authors.fullname: Yunzhi Zhang
paper_authors.type: user
paper_authors.user: Yunzhi
paper_authors._id: 653f2185b424289c5f839a13
paper_authors.name: Eric Ryan Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-30T13:38:41.553Z
paper_authors._id: 6338dd0db7ce9192552cdf7f
paper_authors.avatarUrl: /avatars/d0080d797490926c53690c1b7602d232.svg
paper_authors.fullname: Eric Chan
paper_authors.type: user
paper_authors.user: ericryanchan
paper_authors._id: 653f2185b424289c5f839a14
paper_authors.name: Dmitry Lagun
paper_authors._id: 653f2185b424289c5f839a15
paper_authors.name: Li Fei-Fei
paper_authors._id: 653f2185b424289c5f839a16
paper_authors.name: Deqing Sun
paper_authors._id: 653f2185b424289c5f839a17
paper_authors.name: Jiajun Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-21T09:32:01.925Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_title: ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image
paper_summary: We introduce a 3D-aware diffusion model, ZeroNVS, for single-image novel view
  synthesis for in-the-wild scenes. While existing methods are designed for
  single objects with masked backgrounds, we propose new techniques to address
  challenges introduced by in-the-wild multi-object scenes with complex
  backgrounds. Specifically, we train a generative prior on a mixture of data
  sources that capture object-centric, indoor, and outdoor scenes. To address
  issues from data mixture such as depth-scale ambiguity, we propose a novel
  camera conditioning parameterization and normalization scheme. Further, we
  observe that Score Distillation Sampling (SDS) tends to truncate the
  distribution of complex backgrounds during distillation of 360-degree scenes,
  and propose "SDS anchoring" to improve the diversity of synthesized novel
  views. Our model sets a new state-of-the-art result in LPIPS on the DTU dataset
  in the zero-shot setting, even outperforming methods specifically trained on
  DTU. We further adapt the challenging Mip-NeRF 360 dataset as a new benchmark
  for single-image novel view synthesis, and demonstrate strong performance in
  this setting. Our code and data are at http://kylesargent.github.io/zeronvs/
paper_ai_summary: A 3D-aware diffusion model, ZeroNVS, addresses challenges in single-image novel view synthesis for complex, in-the-wild scenes, setting new benchmarks and improving diversity in synthesized views.
paper_ai_keywords: 3D-aware diffusion model novel view synthesis data mixture depth-scale ambiguity camera conditioning normalization scheme Score Distillation Sampling SDS anchoring LPIPS DTU dataset Mip-NeRF 360 dataset