title: SIGNeRF: Scene Integrated Generation for Neural Radiance Fields
summary: Advances in image diffusion models have recently led to notable improvements
  in the generation of high-quality images. In combination with Neural Radiance
  Fields (NeRFs), they enabled new opportunities in 3D generation. However, most
  generative 3D approaches are object-centric and applying them to editing
  existing photorealistic scenes is not trivial. We propose SIGNeRF, a novel
  approach for fast and controllable NeRF scene editing and scene-integrated
  object generation. A new generative update strategy ensures 3D consistency
  across the edited images, without requiring iterative optimization. We find
  that depth-conditioned diffusion models inherently possess the capability to
  generate 3D consistent views by requesting a grid of images instead of single
  views. Based on these insights, we introduce a multi-view reference sheet of
  modified images. Our method updates an image collection consistently based on
  the reference sheet and refines the original NeRF with the newly generated
  image set in one go. By exploiting the depth conditioning mechanism of the
  image diffusion model, we gain fine control over the spatial location of the
  edit and enforce shape guidance by a selected region or an external mesh.
paper_authors._id: 65962815aa677adc97dc3c2d
paper_authors.name: Jan-Niklas Dihlmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:35:20.775Z
paper_authors._id: 631b6dbcbf1351ed2bd05be1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631b6dbcbf1351ed2bd05be1/_kzcDD4qCifsb-651_LmY.png
paper_authors.fullname: Jan-Niklas Dihlmann
paper_authors.type: user
paper_authors.user: JDihlmann
paper_authors._id: 65962815aa677adc97dc3c2e
paper_authors.name: Andreas Engelhardt
paper_authors._id: 65962815aa677adc97dc3c2f
paper_authors.name: Hendrik Lensch
paper_title: SIGNeRF: Scene Integrated Generation for Neural Radiance Fields
paper_summary: Advances in image diffusion models have recently led to notable improvements
  in the generation of high-quality images. In combination with Neural Radiance
  Fields (NeRFs), they enabled new opportunities in 3D generation. However, most
  generative 3D approaches are object-centric and applying them to editing
  existing photorealistic scenes is not trivial. We propose SIGNeRF, a novel
  approach for fast and controllable NeRF scene editing and scene-integrated
  object generation. A new generative update strategy ensures 3D consistency
  across the edited images, without requiring iterative optimization. We find
  that depth-conditioned diffusion models inherently possess the capability to
  generate 3D consistent views by requesting a grid of images instead of single
  views. Based on these insights, we introduce a multi-view reference sheet of
  modified images. Our method updates an image collection consistently based on
  the reference sheet and refines the original NeRF with the newly generated
  image set in one go. By exploiting the depth conditioning mechanism of the
  image diffusion model, we gain fine control over the spatial location of the
  edit and enforce shape guidance by a selected region or an external mesh.
paper_ai_summary: SIGNeRF, a novel method for NeRF scene editing and object generation, utilizes depth-conditioned diffusion models for 3D consistent image updates without iterative optimization.
paper_ai_keywords: Neural Radiance Fields NeRF depth-conditioned diffusion models 3D generation scene editing multi-view reference sheet shape guidance external mesh