title: Decompositional Neural Scene Reconstruction with Generative Diffusion
    Prior
summary: Decompositional reconstruction of 3D scenes, with complete shapes and
  detailed texture of all objects within, is intriguing for downstream
  applications but remains challenging, particularly with sparse views as input.
  Recent approaches incorporate semantic or geometric regularization to address
  this issue, but they suffer significant degradation in underconstrained areas
  and fail to recover occluded regions. We argue that the key to solving this
  problem lies in supplementing missing information for these areas. To this end,
  we propose DP-Recon, which employs diffusion priors in the form of Score
  Distillation Sampling (SDS) to optimize the neural representation of each
  individual object under novel views. This provides additional information for
  the underconstrained areas, but directly incorporating diffusion prior raises
  potential conflicts between the reconstruction and generative guidance.
  Therefore, we further introduce a visibility-guided approach to dynamically
  adjust the per-pixel SDS loss weights. Together these components enhance both
  geometry and appearance recovery while remaining faithful to input images.
  Extensive experiments across Replica and ScanNet++ demonstrate that our method
  significantly outperforms SOTA methods. Notably, it achieves better object
  reconstruction under 10 views than the baselines under 100 views. Our method
  enables seamless text-based editing for geometry and appearance through SDS
  optimization and produces decomposed object meshes with detailed UV maps that
  support photorealistic Visual effects (VFX) editing. The project page is
  available at https://dp-recon.github.io/.
paper_authors._id: 67dc367858db060fe5a82916
paper_authors.name: Junfeng Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:41:52.218Z
paper_authors._id: 65ae5edddacd99fd58277620
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ae5edddacd99fd58277620/Yq2adEKtkxaSzdEhCfpnY.jpeg
paper_authors.fullname: Junfeng Ni
paper_authors.type: user
paper_authors.user: JunfengNi
paper_authors._id: 67dc367858db060fe5a82917
paper_authors.name: Yu Liu
paper_authors._id: 67dc367858db060fe5a82918
paper_authors.name: Ruijie Lu
paper_authors._id: 67dc367858db060fe5a82919
paper_authors.name: Zirui Zhou
paper_authors._id: 67dc367858db060fe5a8291a
paper_authors.name: Song-Chun Zhu
paper_authors._id: 67dc367858db060fe5a8291b
paper_authors.name: Yixin Chen
paper_authors._id: 67dc367858db060fe5a8291c
paper_authors.name: Siyuan Huang
paper_title: Decompositional Neural Scene Reconstruction with Generative Diffusion
    Prior
paper_summary: Decompositional reconstruction of 3D scenes, with complete shapes and
  detailed texture of all objects within, is intriguing for downstream
  applications but remains challenging, particularly with sparse views as input.
  Recent approaches incorporate semantic or geometric regularization to address
  this issue, but they suffer significant degradation in underconstrained areas
  and fail to recover occluded regions. We argue that the key to solving this
  problem lies in supplementing missing information for these areas. To this end,
  we propose DP-Recon, which employs diffusion priors in the form of Score
  Distillation Sampling (SDS) to optimize the neural representation of each
  individual object under novel views. This provides additional information for
  the underconstrained areas, but directly incorporating diffusion prior raises
  potential conflicts between the reconstruction and generative guidance.
  Therefore, we further introduce a visibility-guided approach to dynamically
  adjust the per-pixel SDS loss weights. Together these components enhance both
  geometry and appearance recovery while remaining faithful to input images.
  Extensive experiments across Replica and ScanNet++ demonstrate that our method
  significantly outperforms SOTA methods. Notably, it achieves better object
  reconstruction under 10 views than the baselines under 100 views. Our method
  enables seamless text-based editing for geometry and appearance through SDS
  optimization and produces decomposed object meshes with detailed UV maps that
  support photorealistic Visual effects (VFX) editing. The project page is
  available at https://dp-recon.github.io/.
paper_ai_summary: DP-Recon uses diffusion priors with Score Distillation Sampling to enhance 3D scene reconstruction from sparse views, recovering underconstrained areas more accurately than existing methods and enabling text-based editing.
paper_ai_keywords: diffusion priors Score Distillation Sampling DP-Recon visibility-guided approach SDS loss weights geometric reconstruction appearance recovery Replica ScanNet++ photorealistic Visual effects (VFX)