title: Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
summary: The correct insertion of virtual objects in images of real-world scenes
  requires a deep understanding of the scene's lighting, geometry and materials,
  as well as the image formation process. While recent large-scale diffusion
  models have shown strong generative and inpainting capabilities, we find that
  current models do not sufficiently "understand" the scene shown in a single
  picture to generate consistent lighting effects (shadows, bright reflections,
  etc.) while preserving the identity and details of the composited object. We
  propose using a personalized large diffusion model as guidance to a physically
  based inverse rendering process. Our method recovers scene lighting and
  tone-mapping parameters, allowing the photorealistic composition of arbitrary
  virtual objects in single frames or videos of indoor or outdoor scenes. Our
  physically based pipeline further enables automatic materials and tone-mapping
  refinement.
paper_authors._id: 66c419b0d6719dab3eebcd9b
paper_authors.name: Ruofan Liang
paper_authors._id: 66c419b0d6719dab3eebcd9c
paper_authors.name: Zan Gojcic
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-26T20:50:25.915Z
paper_authors._id: 6366cda3361a96184dc22139
paper_authors.avatarUrl: /avatars/d8a88c84cb5f69e69dd038674a29be89.svg
paper_authors.fullname: Zan Gojcic
paper_authors.type: user
paper_authors.user: zgojcic
paper_authors._id: 66c419b0d6719dab3eebcd9d
paper_authors.name: Merlin Nimier-David
paper_authors._id: 66c419b0d6719dab3eebcd9e
paper_authors.name: David Acuna
paper_authors._id: 66c419b0d6719dab3eebcd9f
paper_authors.name: Nandita Vijaykumar
paper_authors._id: 66c419b0d6719dab3eebcda0
paper_authors.name: Sanja Fidler
paper_authors._id: 66c419b0d6719dab3eebcda1
paper_authors.name: Zian Wang
paper_title: Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
paper_summary: The correct insertion of virtual objects in images of real-world scenes
  requires a deep understanding of the scene's lighting, geometry and materials,
  as well as the image formation process. While recent large-scale diffusion
  models have shown strong generative and inpainting capabilities, we find that
  current models do not sufficiently "understand" the scene shown in a single
  picture to generate consistent lighting effects (shadows, bright reflections,
  etc.) while preserving the identity and details of the composited object. We
  propose using a personalized large diffusion model as guidance to a physically
  based inverse rendering process. Our method recovers scene lighting and
  tone-mapping parameters, allowing the photorealistic composition of arbitrary
  virtual objects in single frames or videos of indoor or outdoor scenes. Our
  physically based pipeline further enables automatic materials and tone-mapping
  refinement.
paper_ai_summary: A personalized diffusion model guides a physically based inverse rendering process to enhance the photorealistic insertion of virtual objects into real-world scenes.
paper_ai_keywords: diffusion models inverse rendering physically based rendering scene lighting tone-mapping materials refinement