title: InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes
summary: We introduce InseRF, a novel method for generative object insertion in the
  NeRF reconstructions of 3D scenes. Based on a user-provided textual description
  and a 2D bounding box in a reference viewpoint, InseRF generates new objects in
  3D scenes. Recently, methods for 3D scene editing have been profoundly
  transformed, owing to the use of strong priors of text-to-image diffusion
  models in 3D generative modeling. Existing methods are mostly effective in
  editing 3D scenes via style and appearance changes or removing existing
  objects. Generating new objects, however, remains a challenge for such methods,
  which we address in this study. Specifically, we propose grounding the 3D
  object insertion to a 2D object insertion in a reference view of the scene. The
  2D edit is then lifted to 3D using a single-view object reconstruction method.
  The reconstructed object is then inserted into the scene, guided by the priors
  of monocular depth estimation methods. We evaluate our method on various 3D
  scenes and provide an in-depth analysis of the proposed components. Our
  experiments with generative insertion of objects in several 3D scenes indicate
  the effectiveness of our method compared to the existing methods. InseRF is
  capable of controllable and 3D-consistent object insertion without requiring
  explicit 3D information as input. Please visit our project page at
  https://mohamad-shahbazi.github.io/inserf.
paper_authors._id: 659f66569f682a3147bd7eb5
paper_authors.name: Mohamad Shahbazi
paper_authors._id: 659f66569f682a3147bd7eb6
paper_authors.name: Liesbeth Claessens
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T11:04:36.881Z
paper_authors._id: 64aec8786eb97a9d93a6c9e3
paper_authors.avatarUrl: /avatars/89a79e1e894a3bd962e3ab0f19a23ad6.svg
paper_authors.fullname: Liesbeth Claessens
paper_authors.type: user
paper_authors.user: CalishaMienona
paper_authors._id: 659f66569f682a3147bd7eb7
paper_authors.name: Michael Niemeyer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T11:05:06.967Z
paper_authors._id: 63f4f16171a5d395c71d2d1a
paper_authors.avatarUrl: /avatars/0377c4e6b35f8bd1a29d38c99a5943c1.svg
paper_authors.fullname: Michael Niemeyer
paper_authors.type: user
paper_authors.user: mniemeyer
paper_authors._id: 659f66569f682a3147bd7eb8
paper_authors.name: Edo Collins
paper_authors._id: 659f66569f682a3147bd7eb9
paper_authors.name: Alessio Tonioni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-11T11:05:17.664Z
paper_authors._id: 63402b30e670ff9cf63d8caa
paper_authors.avatarUrl: /avatars/0aee84d132a78d4ec71663836a57a245.svg
paper_authors.fullname: Alessio Tonioni
paper_authors.type: user
paper_authors.user: Alessiot
paper_authors._id: 659f66569f682a3147bd7eba
paper_authors.name: Luc Van Gool
paper_authors._id: 659f66569f682a3147bd7ebb
paper_authors.name: Federico Tombari
paper_title: InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes
paper_summary: We introduce InseRF, a novel method for generative object insertion in the
  NeRF reconstructions of 3D scenes. Based on a user-provided textual description
  and a 2D bounding box in a reference viewpoint, InseRF generates new objects in
  3D scenes. Recently, methods for 3D scene editing have been profoundly
  transformed, owing to the use of strong priors of text-to-image diffusion
  models in 3D generative modeling. Existing methods are mostly effective in
  editing 3D scenes via style and appearance changes or removing existing
  objects. Generating new objects, however, remains a challenge for such methods,
  which we address in this study. Specifically, we propose grounding the 3D
  object insertion to a 2D object insertion in a reference view of the scene. The
  2D edit is then lifted to 3D using a single-view object reconstruction method.
  The reconstructed object is then inserted into the scene, guided by the priors
  of monocular depth estimation methods. We evaluate our method on various 3D
  scenes and provide an in-depth analysis of the proposed components. Our
  experiments with generative insertion of objects in several 3D scenes indicate
  the effectiveness of our method compared to the existing methods. InseRF is
  capable of controllable and 3D-consistent object insertion without requiring
  explicit 3D information as input. Please visit our project page at
  https://mohamad-shahbazi.github.io/inserf.
paper_ai_keywords: NeRF InseRF generative object insertion 3D scenes textual description 2D bounding box diffusion models 3D scene editing text-to-image strong priors style and appearance changes monocular depth estimation Tensor-Web Compiling Co-embedding single-view object reconstruction 3D consistency Hugging Face