## Localized Gaussian Splatting Editing with Contextual Awareness

Hanyuan Xiao 1,2 Yingshu Chen 3 Huajian Huang 3 Haolin Xiong 4 Jing Yang 1,2 Pratusha Prasad 1,2 Yajie Zhao 1,2, *

1 University of Southern California 2 Institute for Creative Technologies 3 HKUST 4 University of California, Los Angeles

{ jyang, hxiao, bprasad, zhao @ict.usc.edu } { ychengw, hhuangbg @connect.ust.hk } { xiongh @ucla.edu }

Localize

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-f35169c2-fd4a-47a0-a75e-26f35f095e62.jpg)

"plant inapot

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-86a56e1f-ccfb-4974-94af-e08501d2c146.jpg)

"vase"

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-0dd395c8-2f41-4a59-9001-378e9f1148a1.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-aa008529-bb73-46b6-9fd4-dd7c88a8fb66.jpg)

## Generate &amp; Replace

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-e4b7502a-8f7b-49a3-91e8-4806cd988a45.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-cd626098-7792-419c-b099-6fb6e5dc0957.jpg)

"a turtle"

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-6d9073a2-ec73-4717-8979-9d7afa5d69a8.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-0e4b129c-9460-4252-a7fe-38b397f44bf4.jpg)

"aknight statue"

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-c683c631-ee70-4c68-95b4-d583cc906d74.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-7cc5e6e5-cdc6-406e-a429-63ed0eb5d791.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-f28f5b10-121e-4c11-a773-9e426244f879.jpg)

"alidded stone pot"

"abuddha statue"

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-6a3d7070-a6c3-4a2a-b98c-c5aca7fa6037.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-d3e3f3eb-24c3-4b4c-b709-fa3ebad40276.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-c2bd435b-449d-4ae7-b0ad-6428293e770b.jpg)

"pumpkin"

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-464954c8-5264-4091-b430-b12879181426.jpg)

"a metal ballon dog"

astronaut "an

Figure 1. We present a novel pipeline for text-guided localized Gaussian splatting scene editing. It replaces and edits regions of interest based on user text inputs, achieving realistic and high quality visual results that naturally match the original context, including illumination and occlusions. Compared to previous methods, we are the first to enable object replacement with consistent illumination to the surroundings.

## Abstract

Recent text-guided generation of individual 3D objects has achieved great success using diffusion priors. However, when directly applied to scene editing tasks (e.g., object replacement, object insertion), these methods lack the context of global information, leading to inconsistencies in il-

* Corresponding Author

lumination or unrealistic occlusion. To bridge the gap, we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS) representation. Our key observation is that inpainting by the state-of-the-art conditional 2D diffusion model [57] excels at filling missing regions while considering the context. In this paper, we introduce the success of 2D diffusion inpainting to 3D scene object replacement. Specifically, our approach automati-

cally picks the representative views that best express the global illumination, and these views are inpainted using the state-of-the-art 2D diffusion model. We further developed a coarse-to-fine optimization pipeline that takes the inpainted representative views as input and converts them into 3D Gaussian splatting to achieve scene editing. To acquire an ideal inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a single view that best represents the scene illumination in target region. In the coarseto-fine 3D lifting component, we first achieve image-to-3D lifting given an ideal inpainted view. In the fine step of Texture Enhancement, we introduce a novel Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and texture details with the inpainting diffusion prior, beyond the scope of the 3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only provides fine-grained texture enhancement, but also urges optimization to respect scene lighting. Our approach efficiently achieves local editing with global illumination consistency and we demonstrate robustness of our method by evaluating editing in real scenes containing explicit highlight and shadows, and compare against the state-of-the-art text-to-3D editing methods.

## 1. Introduction

Recent advancements in text-guided 3D object generation have shown remarkable success by leveraging the power of diffusion models [25, 28, 38, 49, 53, 54]. However, little attention has been paid to text-guided localized scene editing task. Naive insertion of generated 3D objects leads to mismatch of illumination appeared in the scene. Global editing methods that do not include localization module [9,14,52] fail by adding color clue to the whole scene or changing undesired regions. Existing solutions for text-guided 3D editing [1,11,48,59] can only achieve minor alternation in geometry or texture of existing objects. Tasks such as context-aware object insertion and object replacement with pronounced change are still underexplored.

In this work, 3D Gaussian Splatting (3DGS) [18] represents both scene and generated object, because 1) it supports efficient optimization in 3D lifting task, and 2) provides explicit representation that enables editing independent of background, in contrast to NeRF representation [34]. To achieve holistically photorealistic 3D editing, it is necessary to take global information such as illumination, occlusions into account. Conventional approaches involving inverse rendering, material generation and physically-based neural rendering are time consuming and unstable [5,24]. Instead, we observed large-scale diffusion models conditioned on background and geometry information (e.g., depth or camera pose) excel at the inpainting task, particularly in regards to maintaining illumination matching and seamlessly fus- ing the generated foreground and background with the original image. Building upon this observation, we introduce a 3D scene editing pipeline that considers scene illumination. By advancing algorithm of 3D lifting and score distillation sampling (SDS) [38], our pipeline accomplishes challenging object insertion and replacement by incorporating prior knowledge from both a depth-guided inpainting diffusion model [57] and a 3D-aware diffusion model [28].

For robust and harmonious object synthesis, we introduce Anchor View Proposal (AVP) algorithm, which selects a view that best represents global illumination with the most contrast. This picked view will be inpainted given the input text prompt, working as the initial guidance for the following two-step contextual 3D lifting. By analysis, this inpainted guidance offers heuristic information for illumination-aware object synthesis. Among the two steps, we first conduct a coarse image-to-3D generation and then an illumination-aware texture enhancement optimization. In the coarse step, we explore score distillation of the 3D-aware diffusion prior from the multi-view diffusion model [28] given the foreground object in the inpainted anchor view. This diffusion prior comprehends essential illumination information from anchor view and guarantees multi-view consistency in 3D object synthesis. In the enhancement step, we propose a new score distillation scheme (DI-SDS) that leverages depth-guided inpainting diffusion prior with background context. This strategy yields multiview-consistent and contextually illumination-aware object refinement.

To summarize, our main contributions include:

- 1. We present a comprehensive pipeline to generate objects based on text inputs that seamlessly integrate into a 3D Gaussian Splatting scene. In particular, our method enables objects generation that automatically matches global illumination.
- 2. We introduce Anchor View Proposal algorithm for automatic selection of a representative view that best represents illumination at a target region within a complex scene. The anchor view selection provides features of scene illumination for harmonious object synthesis.
- 3. We propose Depth-guided Inpainting Score Distillation Sampling, which embeds both condition of geometry and contextual illumination for object generation and texture enhancement.

## 2. Related Works

Recent breakthroughs in large-scale vision-language models (VLMs) [23,41] and text-guided generative models featuring Stable Diffusion [42] have boosted the research field in 2D-to-3D generation and editing, leveraging these

Figure 2. Overview of Pipeline . We propose a pipeline for localized scene editing, the new objects match the global illumination in the original scene. Our method uses text prompts to guide localization and generation, enabling us to synthesize and insert an object that is aware of the surrounding lighting. We introduce Anchor View Proposal to find a representative view of the illumination, which enables a two-step 3D lifting pipeline. Specifically, we propose illumination-aware Texture Enhancement with Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which leverages contextual inpainting capability of large diffusion model. This enhancement step not only improves delicate texture details and motifs but also preserves global lighting conditions during optimization.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-7af5d226-a4a5-4175-9411-d0a54f01e427.jpg)

powerful priors. We review recent literature on image- and text-guided 3D generation and 3D scene editing, focusing on scenes using radiance field representation.

## 2.1. 3D Object Generation

Early exploration of text-to-3D utilizes pre-trained VLMs such as CLIP [41] for zero-shot 3D model generation [17, 20, 44], but with simple shape and texture styles. Subsequently, generative diffusion models equipped with advanced neural field representations [55] have demonstrated remarkable capabilities in high-fidelity 3D generation tasks via 2D-to-3D lifting, encompassing text-to3D [16, 25, 28, 32, 38, 49, 53, 54, 58] and image-to-3D domains [27, 30, 39, 49, 58]. Notably, DreamFusion [38] and SJC [53] introduced the Score Distillation Sampling (SDS) optimization scheme by distilling 2D diffusion knowledge, yielding encouraging text-to-3D results. The follow-up works further extended SDS to resolve issues such as low resolution and over-saturation. For example, Magic3D [25] employs a coarse-to-fine strategy using image and latent diffusion priors for high-quality generation, ProlificDreamer [54] addresses issues of over-saturation, oversmoothing, and low diversity through variational score distillation (VSD), and D-SDS [16] enhances SDS by incorporating score debiasing and prompt debiasing mechanisms. In parallel, with a single image input, Zero123 [28] excels in generating realistic novel perspectives via a viewdependent diffusion model, while SyncDreamer [30] synthesizes view-consistent novel perspectives without relying on neural fields through an intermediate synchronization statistics approach. With recent advanced 3D Gaussian Splatting (3DGS) [18] scene representation featuring swift training and rendering speed, [7, 49] increase generation speed and achieve text-to-3D generation with delicate texture details. These works primarily focus on singleobject generation neglecting background environment interactions, such as ambient illumination and atmosphere. In contrast, our approach incorporates background effect into the 3D generation process with holistic contextual harmony between generation and background scene.

## 2.2. 3D Scene Editing

The rising popularity of implicit 3D representations, particularly radiance fields [18,34], has increased interest in 3D scene geometry and texture editing [1,10,26,29,36,50,59]. Advances in this field utilizes pre-trained vision, language and generative models [4,23,41,42,47], and achieves either direct modifications of entire scene [9, 14, 51] or intricate manipulations and compositions of scene elements [1, 12, 21,33,52,59].

Given a textual editing instruction, NeRF-Art [51] streamlines the modification of simple scenes by combining a foundational NeRF model with a style-oriented NeRF steered by CLIP priors. To progressively edit a NeRF scene, Instruct-NeRF2NeRF [14] alternatively updates training multi-views by Instruct-Pix2Pix [3] and then finetunes NeRF scene with updated training views. Later, to address the issue of multi-view inconsistent editing, ViCANeRF [9] enhances this process by integrating multi-view awareness with a blending mechanism. For more intricate scene alterations, SINE [1] advances image-guided 3D semantic editing by using an editing field informed by geometry and semantic vision priors [4, 8] to infuse detailed geometric and textural shifts. Meanwhile, some 3D edit-

ing works [33,45,46,48,59] focus on precise local changes with 3D segmentation and semantic editing. Within a target region in a NeRF scene, [48] achieves both geometry and texture alterations, and [45,46] support object addition. DreamEditor [59] polishes local editing with a mesh-based field distilling from NeRF that can edit specific explicit segments within the 3D landscape. Concurrent works [6, 11] employ 3DGS for swift text-guided scene editing. Compared to these editing works, we resolve the problem of object generation and replacement with dramatic geometry changes and background alignment.

## 3. Preliminaries

## 3.1. Score Distillation Sampling

Text-guided 3D generation has attracted increasing research interest and demonstrated significant progress by optimizing a 3D representation θ using a 2D pre-trained image diffusion prior ϵ ϕ based on Score Distillation Sampling (SDS), as originally proposed in DreamFusion [38]. The diffusion model ϕ is pre-trained to predict sampled noise ϵ ϕ ( x t ; t, y ) at timestep t of the noisy image x t , conditioned on text embeddings y . By rendering a random view by a differentiable renderer g ( ) · , the aim is to minimize the discrepancy between the rendered image x = g θ ( ) and the diffusion model distribution through the SDS loss. The scene parameterized by θ can be updated by computing the gradient derived from SDS loss:

$$\nabla _ { \theta } \mathcal { L } _ { S D S } ( \phi, \mathbf x ) = \mathbb { E } _ { t, \epsilon } [ w ( t ) ( \epsilon _ { \phi } ( x _ { t } ; t, y ) - \epsilon ) \frac { \partial \mathbf x } { \partial \theta } ], \ \ ( 1 ) \quad \text{$E$}$$

where w t ( ) is a weighting function, ϵ is a Gaussian noise. Our method adapted SDS loss to incorporate the 3D-aware diffusion prior and a customized inpainting diffusion prior, compatible with 3D Gaussian Splatting representations.

## 3.2. 3D Gaussian Splatting

3D Gaussian Splatting (3DGS) [18] represents the scene with a set of anisotropic 3D Gaussians G ( x ) which can be efficiently projected into 2D Gaussians G ′ ( x ′ ) for rendering. The differential rasterization process makes training and rendering faster than previous methods [2, 35]. By optimizing the multivariate Gaussians parameterized by position µ , anisotropic covariance Σ = RSS T R T , opacity α i and color c i information via posed multi-view images, 3DGS achieves an accurate representation of the scene with accelerated training and rendering times. The color C ( x ′ ) and depth D ( x ′ ) of each pixel x ′ in the image can be ob- tained via point-based volumetric rendering:

$$\mathbf C ( x ^ { \prime } ) & = \sum _ { i \in \mathcal { N } } c _ { i } \sigma _ { i } \prod _ { j = 1 } ^ { i - 1 } ( 1 - \sigma _ { j } ), \\ \mathbf D ( x ^ { \prime } ) & = \sum _ { i \in \mathcal { N } } d _ { i } \sigma _ { i } \prod _ { j = 1 } ^ { i - 1 } ( 1 - \sigma _ { j } ), \sigma _ { i } = \alpha _ { i } \mathbf G ^ { \prime } ( x ^ { \prime } )$$

where N is ordered points overlapping the pixel, and d i is the distance between 3D Gaussian center and camera center.

## 4. Method

Given an unbounded 3D Gaussian Splatting (3DGS) representation [18] scene, our objective is to perform textguided localized 3D scene editing, specifically object insertion or replacement. We first leverage an off-the-shelf method [40] to localize a 3D bounding box as the target editing region. Then, we sample and render azimuth camera viewpoints around the bounding box to feed into our proposed Anchor View Proposal (AVP) module (Sec. 4.1), which votes for a view to be used in the 3D lifting step. Next, we take a user-specified text prompt ˆ y as guidance to inpaint the anchor view, obtaining an image x inpaint . The object foreground of x inpaint , x fore inpaint , is segmented out by [22] and fed into our coarse-to-fine 3D generation and texture enhancement pipeline (Sec. 4.2). In experiments, we assume the generation objects are opaque. The overview of our pipeline is illustrated in Fig. 2.

## 4.1. Anchor View Proposal

Existing multi-view diffusion models [28, 56] demonstrate the ability to generate consistent illumination across multi-views given the baked-in lighting in input single-view image. Inspired by this observation, we aim to identify an anchor viewpoint around the bounding box that includes the strongest illumination cues, such as shadow and highlight. To this end, we propose an Anchor View Proposal (AVP) algorithm to reliably select a well-conditioned view from rendering of N anc surrounding viewpoints looking at the center of bounding box. The ideal proposed view should contain the most significant difference in illumination between the left-right, top-bottom, or diagonal halves of the rendered images (see evaluation in Sec. 5.2 and Sec. 5.4). This is achieved by rotating each rendered image by a predefined set of angles and selecting the rotation that places majority of brighter pixels to the left. For robust lighting estimation from images, we convert rendered RGB images to HSV color space. This design choice is enlightened by the fact that V is the distance along the axis that turns black into non-black color, also known as 'brightness'. Therefore, computation of V-channel image estimates amount of light that lit the surface. In scenarios where the majority of

illumination is predominantly cast in the top-bottom direction, our algorithm tends to output an arbitrary view. This behavior aligns with our requirement, as we sample viewpoints in azimuth and any view is considered equally suitable. Pseudocode of AVP algorithm is provided in the supplementary material.

## 4.2. Context-aware Coarse-to-Fine 3D Generation

After acquiring the anchor view, we use an off-the-shelf depth-conditioned diffusion model [57] to inpaint the mask of bounding box projection, according to the generation text prompt ˆ y . The foreground served as input to the next 3D lifting pipeline can be extracted by the segmentation model [22]. For efficient and robust 3D generation with contextual illumination awareness, we propose to conduct a coarse but swift image-to-3D generation via 3D-aware diffusion prior, and an illumination-aware texture enhancement step to further realize high-quality 3D lifting result. Note that after removal (if any), we leave the scene points untouched and only optimize new generated points.

## 4.2.1 Coarse Image-to-3D Generation

Recent efficient multi-view diffusion models [28] were pretrained on large-scale 3D objects with backed lighting. Given the lighting-heuristic inpainted anchor view, in the coarse step, the multi-view diffusion model can faithfully lift the inpainted image to 3D with multi-view consistency.

To achieve reliable 3D lifting in 3DGS representation, we adopt compact-based densification and pruning strategy proposed in [7]. Instead of initializing 3D Gaussians from Point-E [37], we initialize them in a solid sphere, with opacity in a normal distribution proportional to the center distances. This strategy often performs better in our coarse step, since the inpainted conditioning image may not align with the generation modality in Point-E, leading to inconsistent points with the input image and longer training time to converge. Still, some initial points may remain as floaters and are hard to be pruned later. Therefore, we significantly lower initial opacity and colors of 3D Gaussians.

During object optimization by randomly sampled views, we aim to minimize the objective function:

$$\mathcal { L } = \lambda _ { r g b } \mathcal { L } _ { r g b } + \lambda _ { m a s k } \mathcal { L } _ { m a s k } + \lambda _ { 3 D - S D S } \mathcal { L } _ { 3 D - S D S }, \ ( 3 ) \quad \nu \quad \nu,$$

where L rgb is the mean squared error (MSE) between the foreground anchor view image x fore inpaint and rendered image at the same viewpoint, L mask is MSE loss between groundtruth mask extracted from x fore inpaint and predicted opacity image m anchor , L 3 D SDS -is score distillation loss (Eqn. 1) from 3D-aware diffusion prior [28] for the sample view. The weight of reference view RGB ( λ rgb ) increases as training proceeds. We compute m anchor by volumetric render- ing equation:

$$\mathbf m _ { a n c h o r } = \sum _ { i \in \mathcal { N } } 1 \cdot \sigma _ { i } \prod _ { j = 1 } ^ { i - 1 } ( 1 - \sigma _ { j } ). \quad \ \ ( 4 )$$

With L mask , we encourage foreground points to be fully opaque and background points to be transparent.

## 4.2.2 Illumination-aware Texture Enhancement

With the first coarse generation step, the synthesized object has a multi-view consistent shape and appearance, but lacks subtle texture details and high resolution required by photorealism. Therefore, we propose a contextually illuminationaware Texture Enhancement step, which not only enriches geometry and texture details, but also preserves multi-view lighting conditions.

While we observe pre-trained diffusion model with SDS (Eqn. 1) can greatly enhance texture and geometry details, it does not receive background information from the scene. Inspired by 2D inpainting, we propose Depth-guided Inpainting Score Distillation Sampling (DI-SDS), where intermediate output features of depth-guided ControlNet copy network C depth , projected bounding box mask and masked image are fed into the diffusion model for conditioned generation. Specifically, we first feed colored image latents x t , timestep t , text prompt embeddings y and depth image d into ControlNet layers C depth to obtain high-level residual block output samples D and middle-level output samples M :

$$( D, M ) = \mathcal { C } _ { d e p t h } ( x _ { t } ; t, y, \mathbf d ), \text{ \quad \ \ } ( 5 )$$

where depth image d = 1 -d (i.e., inversed depth), and d is derived from volumetric rendering process (Eqn. 2) and scaled into range [0 , 1] .

Then, we concatenate bounding box mask m and masked image latents m l with colored image latents, and feed them additionally into 2D diffusion model ϕ with D,M :

$$& \nabla _ { \theta } \mathcal { L } _ { D I - S D S } ( \phi, \mathbf x ) = \\ & \mathbb { E } _ { t, \epsilon } [ w ( t ) ( \epsilon _ { \phi } ( x _ { t }, m, m _ { l } ; t, y, D, M ) - \epsilon ) \frac { \partial \mathbf x } { \partial \theta } ],$$

where θ is 3DGS scene parameters, ϵ ϕ ( ) · pled noise, ϵ is a Gaussian noise, x is predicted samis rendered RGB image.

We accommodate classifier-free guidance by computing predicted noise with the following equation:

$$& \hat { \epsilon _ { \phi } } ( x _ { t }, m, m _ { l } ; t, y, \mathbf d ) = \epsilon _ { \phi } ( x _ { t }, m, m _ { l } ; t, y, \mathbf d ) \\ & + s \cdot ( \epsilon _ { \phi } ( x _ { t }, m, m _ { l } ; t, y, \mathbf d ) - \epsilon _ { \phi } ( x _ { t } ; t ) ),$$

where s is guidance scale specified in Sec. 5.1.

The text prompt guiding generation ˆ y is embedded into view-conditioned text embeddings y , for example, 'a standing pineapple, photorealistic, front view', to prevent Janus

problem and provide multi-view information. Since we optimize both object geometry and texture through finetuning, we lower the learning rate in geometry with a focus on smooth texture refinement. During optimization, we merge Gaussian splats of background color field and generated object splats, and render randomly sampled views following strategy in [13]. In Texture Enhancement step, we disable L 3 D SDS -because it prevents details generated by 2D inpainting diffusion model that does not share the same prior distribution. Therefore, the loss function of this step becomes

$$\mathcal { L } = \lambda _ { r g b } \mathcal { L } _ { r g b } + \lambda _ { m a s k } \mathcal { L } _ { m a s k } + \lambda _ { D I - S D S } \mathcal { L } _ { D I - S D S }. \ ( 8 )$$

Providing flexible controllability for users, this Texture Enhancement step can be balanced by fewer optimization iterations in some cases when simple geometry and texture patterns are sufficiently satisfactory. For example, the monkey statue is expected to have smooth skin in Fig. 6.

Figure 3. Evaluation on Anchor View Proposal algorithm. We evaluate the robustness of our Anchor View Proposal (AVP) algorithm by comparing view angle differences between human and algorithm proposals on six objects. The top graph shows the Valuechannel (in HSV space) brightness ratio (range: [0 , 1] ) for 100 continuous azimuth views, where a lower value indicates the left side is darker. In this context, A VP selects views with the lowest difference ratio, closely matching human selections. The bottom examples further demonstrate AVP's effectiveness in selecting high contrast views.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-6abb3530-5c01-4841-8587-1216dc0f05e6.jpg)

## 5. Experiment

## 5.1. Implementation Details

Datasets. To evaluate performance in diverse scenes, we use LERF [19], MipNeRF360 [2] and two self-captured

"a square seat cushion"

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-751ce65b-5a56-4505-a17c-c6ef9913ee28.jpg)

Figure 4. Ablation study on Contextual 3D Lifting. We show novel views of localized object editing when we skip coarse step, skip Texture Enhancement step, and apply full pipeline. Coarse step aims to provide initial prior and scene illumination in texture. Texture Enhancement aims to generate more details in geometry and texture.

Figure 5. Ablation study on Anchor View Proposal in terms of influence to image-to-3D result. Top row: proposed anchor view. Bottom row: non-anchor view. We show inpainted images with the same generator seed in the left column and multi-view rendering of 3D lifted object. Conditioning view that does not reflect scene illumination fails to generate texture that fits into the scene from other viewpoints.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-e8f2b166-eb4f-4355-896a-d62ef21fe26a.jpg)

datasets where we set up directional light sources. We focus on scenes with challenging illumination that may include multiple objects to test robustness of our pipeline.

Pre-trained Models. We use released Stable Zero123 model in coarse step, ControlNet Depth version [57] in 2D inpainting and Texture Enhancement step, and Stable Diffusion [43] as 2D diffusion model. Specifically, we leveraged Stable Diffusion Inpainting model by Runway as 2D diffusion model in Texture Enhancement. For view conditioned text embedding, we used Stable Diffusion v1.5 by Runway. Forwarding through ControlNet Depth and Stable Diffusion together provides predicted noise residual. Parameters of all diffusion models are frozen in training.

## 5.2. Qualitative Results

We mix and match text prompt with scenes for generation and render multi-view images for qualitative evaluation in Fig. 6. In the figure, we show the rendering of input scene

Figure 6. Qualitative Results. (a) is input pre-trained 3D Gaussian Splatting scene with bounding box marked in red. (b) is anchor view image with inpainted object according to text prompt on top of each row. (c-f) are multi-view rendering of output scenes. More results are presented in supplementary material.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-5b16d646-eb75-4ffd-9346-0f8b539e981c.jpg)

under anchor view with 3D bounding box visualized, and 2D inpainted object under proposed anchor view. Then, we show four multi-view images rendered from output scene. Our result demonstrates detailed texture and faithful respect to scene illumination.

Evaluation on Anchor View Proposal algorithm. We selected six bounding boxes and their associated objects with strong illumination, then generated multi-view images as input into AVP algorithm. Specifically, we sample 100 object-centered viewpoints along azimuth with every two nearby views are 3.6 degrees apart. Then, we had human manually pick the 'best' view with the strongest highlight or shadow on left or right half depending on requirements. We graph the offset in viewpoint indices as selected by both the human and our algorithm, and present the three pairs of views with the highest offsets in Fig. 3. In the plot, we observe that the difference in brightness between left and right half strongly implies the optimal viewpoint to pick. In the case with the highest offset (i.e., the white sculpture), both the human-picked and the algorithm-proposed views are good enough as conditioning viewpoint.

## 5.3. Comparison

Qualitative Comparison. We compare our method against Vica-NeRF [9] and GaussianEditor [6] by presenting qualitative result in Fig. 7. While we additionally provide localization information in text prompt to Vica-NeRF, it is unable to edit the region of interest or preserve background color. Same as ours, GaussianEditor performs segmentation via SAM and 3D lifting from 2D inpainted im-

"[Put] a teddy bear [on bench]"

"[Put] a pikachu [on chair]"

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00083v1-figure-f2843e65-c123-4ffe-88a2-a6a2454d750d.jpg)

Figure 7. Comparison. We show visual results from ours, GaussianEditor [6] and Vica-NeRF [9], along with a reference image of original scene. Specifically, we provide 1) longer text prompt (with words in square bracket) on top of each row containing localization information to Vica-NeRF; 2) shorter text prompt (without words in square brackets) and pre-determined bounding box to ours and GaussianEditor.

Table 1. CLIP Score ↑ of compared cases in Fig. 7.

## 5.4. Ablation Study

|               |   Vica-NeRF |   GaussianEditor |   Ours |
|---------------|-------------|------------------|--------|
| Teddy Bear    |      0.582  |           0.7461 | 0.7817 |
| Buddha Statue |      0.5596 |           0.7578 | 0.7725 |
| Pikachu       |      0.5898 |           0.7939 | 0.8247 |
| Astronaut     |      0.5518 |           0.6719 | 0.6748 |
| Dinosaur      |      0.6777 |           0.7627 | 0.7354 |
| Sculpture     |      0.5063 |           0.7275 | 0.7104 |

age. However, GaussianEditor neither includes a mechanism to pick views containing illumination information, nor uses background in multi-view optimization [31]. Therefore, it suffers from negligence to the scene illumination such as Pikachu and astronaut cases in Fig. 7.

Quantitative Comparison. We report CLIP Score [15] for quantitative comparisons among baselines in Tab. 1. Specifically, we leverage open-sourced ViT-B/32 model of CLIP. We use foreground image inside mask to filter out influence of background when CLIP evaluates text description. We formulate text prompt by filling sentence 'A photo of {} ', where text prompt in {} can be found in Fig. 7. Result shows that our generated objects are more aligned with input text prompt. In addition, we conducted a user study on the results and collected statistics of user's preference. In the questionnaire, we put outputs of different methods in random order and ask participants which one is the most consistent with the text prompt and looks the most realistic within the scene. Statistics show that 0% users prefer Vica-NeRF, 29.49% prefer GaussianEditor and 70.51% prefer ours.

Effect on Coarse-to-Fine 3D Generation. In Fig. 4, we show multi-view rendering of a generated square cushion on a chair when we use texture enhancement only, coarse step only, and full pipeline. Without coarse step, the generated object suffers from inconsistency in multi-view and weak fitting to scene illumination. Without Texture Enhancement, we observe consistent geometry, smooth texture, and faithful lifting from anchor view image; however, more details in texture can be added while generation is still aligned with text prompt. With both steps, we observe a significant amount of detail added to both geometry and texture while highlight and shadow cast on the surface still respect the scene. We provide average CLIP score of multi-view rendering following the same way in quantitative comparison (Sec. 5.3).

Effect on Anchor View Proposal. In Fig. 5, we show multi-view rendering of coarse step output when we inpaint the proposed anchor view and another non-anchor view with same inpainting seed and input text prompt 'a green ball'. With the proposed inpainted anchor view, our coarse step and 3D-aware diffusion prior are robust enough to generate multi-views in contextually consistent illumination. The anchor view is more suitable for 3D photorealism, as it implies heuristic illumination information.

## 6. Conclusion

In this paper, we proposed an end-to-end pipeline to address text-guided localized editing of 3D scenes, focusing on generation of objects that naturally fit into the scene with illumination awareness. We observed that pre-trained 2D inpainting diffusion model is able to generate objects with texture consistent with scene illumination. Inspired by this, we proposed an Anchor View Proposal algorithm to find a viewpoint, under which we can obtain a conditioning image that contains the most contextual lighting information. We then proposed context-aware coarse-to-fine 3D generation pipeline that faithfully lifts conditioning image into 3D via a coarse step, and then adds details into geometry and texture via a Texture Enhancement step. To resolve the bottleneck that previous SDS does not support optimization with rendering background, we derived DI-SDS from depth inpainting ControlNet to supervise the Texture Enhancement. Experiments demonstrate that our pipeline successfully distills contextual illumination from the input scene.

## Acknowledgement

This research is sponsored by the U.S. Army under contract number W911NF-14-D-0005, Cooperative Agreement number W911NF-20-2-0053 and by the Office of Naval Research under contract number N00014-21-S-SN03. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army, Navy, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

## References

- [1] Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20919-20929, 2023. 2, 3
- [2] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR , 2022. 4, 6
- [3] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1839218402, June 2023. 3
- [4] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e J´ egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021. 3
- [5] Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, and Matthias Nießner. Text2tex: Text-driven texture synthesis via diffusion models. arXiv preprint arXiv:2303.11396 , 2023. 2
- [6] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In CVPR , 2024. 4, 7, 8
- [7] Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3d using gaussian splatting. In IEEE Conference on Computer Vision and Pattern Recognition , 2024. 3, 5
- [8] Yu Deng, Jiaolong Yang, and Xin Tong. Deformed implicit field: Modeling 3d shapes with learned dense correspondence. In IEEE Computer Vision and Pattern Recognition , 2021. 3
- [9] Jiahua Dong and Yu-Xiong Wang. ViCA-neRF: Viewconsistency-aware 3d editing of neural radiance fields. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 2, 3, 7, 8
- [10] Zhiwen Fan, Yifan Jiang, Peihao Wang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Unified implicit neural stylization. In European Conference on Computer Vision , 2022. 3
- [11] Jiemin Fang, Junjie Wang, Xiaopeng Zhang, Lingxi Xie, and Qi Tian. Gaussianeditor: Editing 3d gaussians delicately with text instructions. In CVPR , 2024. 2, 4
- [12] Ori Gordon, Omri Avrahami, and Dani Lischinski. Blendednerf: Zero-shot object generation and blending in existing neural radiance fields. In International Conference on Computer Vision AI3DCC Workshop , 2023. 3
- [13] Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, ZiXin Zou, Chen Wang, Yan-Pei Cao, and Song-Hai Zhang. threestudio: A unified framework for 3d content generation. https://github.com/threestudio-project/ threestudio , 2023. 6
- [14] Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2nerf: Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023. 2, 3
- [15] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: a reference-free evaluation metric for image captioning. In EMNLP , 2021. 8
- [16] Susung Hong, Donghoon Ahn, and Seungryong Kim. Debiasing scores and prompts of 2d diffusion for view-consistent text-to-3d generation. Advances in Neural Information Processing Systems , 36, 2024. 3
- [17] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 867-876, 2022. 3
- [18] Kerbl, Bernhard and Kopanas, Georgios, Thomas Leimk¨ uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), July 2023. 2, 3, 4

- [19] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In International Conference on Computer Vision (ICCV) , 2023. 6
- [20] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu. Clip-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers , December 2022. 3
- [21] Hyunsu Kim, Gayoung Lee, Yunjey Choi, Jin-Hwa Kim, and Jun-Yan Zhu. 3d-aware blending with generative nerfs. In ICCV , 2023. 3
- [22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll´ ar, and Ross Girshick. Segment anything, 2023. 4, 5
- [23] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , pages 1288812900. PMLR, 2022. 2, 3
- [24] Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. Gs-ir: 3d gaussian splatting for inverse rendering. arXiv preprint arXiv:2311.16473 , 2023. 2
- [25] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 2, 3
- [26] Kunhao Liu, Fangneng Zhan, Yiwen Chen, Jiahui Zhang, Yingchen Yu, Abdulmotaleb El Saddik, Shijian Lu, and Eric P Xing. Stylerf: Zero-shot 3d style transfer of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 83388348, 2023. 3
- [27] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without pershape optimization. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 3
- [28] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9298-9309, 2023. 2, 3, 4, 5
- [29] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5773-5783, 2021. 3
- [30] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. In The Twelfth International Conference on Learning Representations , 2024. 3
- [31] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, and Wenping Wang.
- Wonder3d: Single image to 3d using cross-domain diffusion, 2023. 8
- [32] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, MingYu Liu, Sanja Fidler, and James Lucas. Att3d: Amortized text-to-3d object synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023. 3
- [33] Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. OBJECT 3DIT: Language-guided 3d-aware image editing. In Thirtyseventh Conference on Neural Information Processing Systems , 2023. 3, 4
- [34] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021. 2, 3
- [35] Thomas M¨ uller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 41(4):102:1102:15, July 2022. 4
- [36] Thu Nguyen-Phuoc, Feng Liu, and Lei Xiao. Snerf: stylized neural implicit representations for 3d scenes. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2022 , 2022. 3
- [37] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts, 2022. 5
- [38] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations , 2023. 2, 3, 4
- [39] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin-Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Bernard Ghanem. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors, 2023. 3
- [40] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In CVPR , 2024. 4
- [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 2, 3
- [42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022. 2, 3
- [43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10684-10695, June 2022. 6

- [44] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, Marco Fumero, and Kamal Rahimi Malekshan. Clip-forge: Towards zero-shot text-to-shape generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18603-18613, 2022. 3
- [45] Mohamad Shahbazi, Liesbeth Claessens, Michael Niemeyer, Edo Collins, Alessio Tonioni, Luc Van Gool, and Federico Tombari. Inserf: Text-driven generative object insertion in neural 3d scenes. Arxiv , 2024. 4
- [46] Ka Chun Shum, Jaeyeon Kim, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung. Language-driven object fusion into neural radiance fields with pose-conditioned dataset updates. In IEEE Conference on Computer Vision and Pattern Recognition , 2024. 4
- [47] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015. 3
- [48] Hyeonseop Song, Seokhun Choi, Hoseok Do, Chul Lee, and Taehyeong Kim. Blending-nerf: Text-driven localized editing in neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14383-14393, 2023. 2, 4
- [49] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In The Twelfth International Conference on Learning Representations , 2024. 2, 3
- [50] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. 2022 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3825-3834, 2021. 3
- [51] Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Nerf-art: Text-driven neural radiance fields stylization. IEEE Transactions on Visualization and Computer Graphics , 2023. 3
- [52] Dongqing Wang, Tong Zhang, Alaa Abboud, and Sabine S¨ usstrunk. Inpaintnerf360: Text-guided 3d inpainting on unbounded neural radiance fields, 2023. 2, 3
- [53] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12619-12629, 2023. 2, 3
- [54] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. 2, 3
- [55] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum , 2022. 3
- [56] Jianglong Ye, Peng Wang, Kejie Li, Yichun Shi, and Heng Wang. Consistent-1-to-3: Consistent image to 3d view syn-
- thesis via geometry-aware diffusion models. In 3DV , 2024. 4
- [57] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023. 1, 2, 5, 6
- [58] Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, BIN FU, Tao Chen, Gang YU, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation. In Thirtyseventh Conference on Neural Information Processing Systems , 2023. 3
- [59] Jingyu Zhuang, Chen Wang, Liang Lin, Lingjie Liu, and Guanbin Li. Dreameditor: Text-driven 3d scene editing with neural fields. In SIGGRAPH Asia 2023 Conference Papers , pages 1-10, 2023. 2, 3, 4
Hanyuan Xiao Yingshu Chen Huajian Huang Haolin Xiong Jing Yang Pratusha Prasad Yajie Zhao
Localized Gaussian Splatting Editing with Contextual Awareness
Recent text-guided generation of individual 3D object has achieved great
success using diffusion priors. However, these methods are not suitable for
object insertion and replacement tasks as they do not consider the background,
leading to illumination mismatches within the environment. To bridge the gap,
we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian
Splatting (3DGS) representation. Our key observation is that inpainting by the
state-of-the-art conditional 2D diffusion model is consistent with background
in lighting. To leverage the prior knowledge from the well-trained diffusion
models for 3D object generation, our approach employs a coarse-to-fine
objection optimization pipeline with inpainted views. In the first coarse step,
we achieve image-to-3D lifting given an ideal inpainted view. The process
employs 3D-aware diffusion prior from a view-conditioned diffusion model, which
preserves illumination present in the conditioning image. To acquire an ideal
inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a
single view that best represents the scene illumination in target region. In
the second Texture Enhancement step, we introduce a novel Depth-guided
Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and
texture details with the inpainting diffusion prior, beyond the scope of the
3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only
provides fine-grained texture enhancement, but also urges optimization to
respect scene lighting. Our approach efficiently achieves local editing with
global illumination consistency without explicitly modeling light transport. We
demonstrate robustness of our method by evaluating editing in real scenes
containing explicit highlight and shadows, and compare against the
state-of-the-art text-to-3D editing methods.