InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes
We introduce InseRF, a novel method for generative object insertion in the
NeRF reconstructions of 3D scenes. Based on a user-provided textual description
and a 2D bounding box in a reference viewpoint, InseRF generates new objects in
3D scenes. Recently, methods for 3D scene editing have been profoundly
transformed, owing to the use of strong priors of text-to-image diffusion
models in 3D generative modeling. Existing methods are mostly effective in
editing 3D scenes via style and appearance changes or removing existing
objects. Generating new objects, however, remains a challenge for such methods,
which we address in this study. Specifically, we propose grounding the 3D
object insertion to a 2D object insertion in a reference view of the scene. The
2D edit is then lifted to 3D using a single-view object reconstruction method.
The reconstructed object is then inserted into the scene, guided by the priors
of monocular depth estimation methods. We evaluate our method on various 3D
scenes and provide an in-depth analysis of the proposed components. Our
experiments with generative insertion of objects in several 3D scenes indicate
the effectiveness of our method compared to the existing methods. InseRF is
capable of controllable and 3D-consistent object insertion without requiring
explicit 3D information as input. Please visit our project page at
https://mohamad-shahbazi.github.io/inserf.
659f66569f682a3147bd7eb5
Mohamad Shahbazi
659f66569f682a3147bd7eb6
Liesbeth Claessens
admin_assigned
2024-01-11T11:04:36.881Z
64aec8786eb97a9d93a6c9e3
/avatars/89a79e1e894a3bd962e3ab0f19a23ad6.svg
Liesbeth Claessens
user
CalishaMienona
659f66569f682a3147bd7eb7
Michael Niemeyer
admin_assigned
2024-01-11T11:05:06.967Z
63f4f16171a5d395c71d2d1a
/avatars/0377c4e6b35f8bd1a29d38c99a5943c1.svg
Michael Niemeyer
user
mniemeyer
659f66569f682a3147bd7eb8
Edo Collins
659f66569f682a3147bd7eb9
Alessio Tonioni
admin_assigned
2024-01-11T11:05:17.664Z
63402b30e670ff9cf63d8caa
/avatars/0aee84d132a78d4ec71663836a57a245.svg
Alessio Tonioni
user
Alessiot
659f66569f682a3147bd7eba
Luc Van Gool
659f66569f682a3147bd7ebb
Federico Tombari
InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes
We introduce InseRF, a novel method for generative object insertion in the
NeRF reconstructions of 3D scenes. Based on a user-provided textual description
and a 2D bounding box in a reference viewpoint, InseRF generates new objects in
3D scenes. Recently, methods for 3D scene editing have been profoundly
transformed, owing to the use of strong priors of text-to-image diffusion
models in 3D generative modeling. Existing methods are mostly effective in
editing 3D scenes via style and appearance changes or removing existing
objects. Generating new objects, however, remains a challenge for such methods,
which we address in this study. Specifically, we propose grounding the 3D
object insertion to a 2D object insertion in a reference view of the scene. The
2D edit is then lifted to 3D using a single-view object reconstruction method.
The reconstructed object is then inserted into the scene, guided by the priors
of monocular depth estimation methods. We evaluate our method on various 3D
scenes and provide an in-depth analysis of the proposed components. Our
experiments with generative insertion of objects in several 3D scenes indicate
the effectiveness of our method compared to the existing methods. InseRF is
capable of controllable and 3D-consistent object insertion without requiring
explicit 3D information as input. Please visit our project page at
https://mohamad-shahbazi.github.io/inserf.
InseRF inserts new 3D objects into scenes based on textual descriptions and 2D bounding boxes, using diffusion models for text-to-image generation and monocular depth estimation.
NeRF InseRF generative object insertion 3D scene editing text-to-image diffusion models 3D generative modeling single-view object reconstruction monocular depth estimation