title: Taming Latent Diffusion Model for Neural Radiance Field Inpainting
summary: Neural Radiance Field (NeRF) is a representation for 3D reconstruction from
  multi-view images. Despite some recent work showing preliminary success in
  editing a reconstructed NeRF with diffusion prior, they remain struggling to
  synthesize reasonable geometry in completely uncovered regions. One major
  reason is the high diversity of synthetic contents from the diffusion model,
  which hinders the radiance field from converging to a crisp and deterministic
  geometry. Moreover, applying latent diffusion models on real data often yields
  a textural shift incoherent to the image condition due to auto-encoding errors.
  These two problems are further reinforced with the use of pixel-distance
  losses. To address these issues, we propose tempering the diffusion model's
  stochasticity with per-scene customization and mitigating the textural shift
  with masked adversarial training. During the analyses, we also found the
  commonly used pixel and perceptual losses are harmful in the NeRF inpainting
  task. Through rigorous experiments, our framework yields state-of-the-art NeRF
  inpainting results on various real-world scenes. Project page:
  https://hubert0527.github.io/MALD-NeRF
paper_authors._id: 661dec6bbca1a6038b587775
paper_authors.name: Chieh Hubert Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:11:30.943Z
paper_authors._id: 62cb9c172e4fbd838f6b4f37
paper_authors.avatarUrl: /avatars/ea0f7d9e746bb8a6e574854deea4859a.svg
paper_authors.fullname: Hubert Lin
paper_authors.type: user
paper_authors.user: hubert0527
paper_authors._id: 661dec6bbca1a6038b587776
paper_authors.name: Changil Kim
paper_authors._id: 661dec6bbca1a6038b587777
paper_authors.name: Jia-Bin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:12:39.974Z
paper_authors._id: 641c139b73296f7ee256970c
paper_authors.avatarUrl: /avatars/5a2550d95e686640242840ad3bd0e680.svg
paper_authors.fullname: Jiabin Huang
paper_authors.type: user
paper_authors.user: YellowAddice
paper_authors._id: 661dec6bbca1a6038b587778
paper_authors.name: Qinbo Li
paper_authors._id: 661dec6bbca1a6038b587779
paper_authors.name: Chih-Yao Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:12:57.881Z
paper_authors._id: 6515bf00c2aedaa7e692a1ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6515bf00c2aedaa7e692a1ec/PYpS1QwTPdQYW5_BrvwzP.png
paper_authors.fullname: Chih-Yao Ma
paper_authors.type: user
paper_authors.user: chihyaoma
paper_authors._id: 661dec6bbca1a6038b58777a
paper_authors.name: Johannes Kopf
paper_authors._id: 661dec6bbca1a6038b58777b
paper_authors.name: Ming-Hsuan Yang
paper_authors._id: 661dec6bbca1a6038b58777c
paper_authors.name: Hung-Yu Tseng
paper_title: Taming Latent Diffusion Model for Neural Radiance Field Inpainting
paper_summary: Neural Radiance Field (NeRF) is a representation for 3D reconstruction from
  multi-view images. Despite some recent work showing preliminary success in
  editing a reconstructed NeRF with diffusion prior, they remain struggling to
  synthesize reasonable geometry in completely uncovered regions. One major
  reason is the high diversity of synthetic contents from the diffusion model,
  which hinders the radiance field from converging to a crisp and deterministic
  geometry. Moreover, applying latent diffusion models on real data often yields
  a textural shift incoherent to the image condition due to auto-encoding errors.
  These two problems are further reinforced with the use of pixel-distance
  losses. To address these issues, we propose tempering the diffusion model's
  stochasticity with per-scene customization and mitigating the textural shift
  with masked adversarial training. During the analyses, we also found the
  commonly used pixel and perceptual losses are harmful in the NeRF inpainting
  task. Through rigorous experiments, our framework yields state-of-the-art NeRF
  inpainting results on various real-world scenes. Project page:
  https://hubert0527.github.io/MALD-NeRF
paper_ai_summary: The proposed framework addresses limitations in NeRF editing by tempering diffusion model stochasticity and mitigating textural shifts, leading to state-of-the-art NeRF inpainting results.
paper_ai_keywords: Neural Radiance Field NeRF diffusion prior latent diffusion models radiance field textural shift auto-encoding errors pixel-distance losses perceptual losses masked adversarial training NeRF inpainting