Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
The correct insertion of virtual objects in images of real-world scenes
requires a deep understanding of the scene's lighting, geometry and materials,
as well as the image formation process. While recent large-scale diffusion
models have shown strong generative and inpainting capabilities, we find that
current models do not sufficiently "understand" the scene shown in a single
picture to generate consistent lighting effects (shadows, bright reflections,
etc.) while preserving the identity and details of the composited object. We
propose using a personalized large diffusion model as guidance to a physically
based inverse rendering process. Our method recovers scene lighting and
tone-mapping parameters, allowing the photorealistic composition of arbitrary
virtual objects in single frames or videos of indoor or outdoor scenes. Our
physically based pipeline further enables automatic materials and tone-mapping
refinement.
66c419b0d6719dab3eebcd9b
Ruofan Liang
66c419b0d6719dab3eebcd9c
Zan Gojcic
claimed_verified
2025-03-26T20:50:25.915Z
6366cda3361a96184dc22139
/avatars/d8a88c84cb5f69e69dd038674a29be89.svg
Zan Gojcic
user
zgojcic
66c419b0d6719dab3eebcd9d
Merlin Nimier-David
66c419b0d6719dab3eebcd9e
David Acuna
66c419b0d6719dab3eebcd9f
Nandita Vijaykumar
66c419b0d6719dab3eebcda0
Sanja Fidler
66c419b0d6719dab3eebcda1
Zian Wang
Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
The correct insertion of virtual objects in images of real-world scenes
requires a deep understanding of the scene's lighting, geometry and materials,
as well as the image formation process. While recent large-scale diffusion
models have shown strong generative and inpainting capabilities, we find that
current models do not sufficiently "understand" the scene shown in a single
picture to generate consistent lighting effects (shadows, bright reflections,
etc.) while preserving the identity and details of the composited object. We
propose using a personalized large diffusion model as guidance to a physically
based inverse rendering process. Our method recovers scene lighting and
tone-mapping parameters, allowing the photorealistic composition of arbitrary
virtual objects in single frames or videos of indoor or outdoor scenes. Our
physically based pipeline further enables automatic materials and tone-mapping
refinement.
A personalized diffusion model guides a physically based inverse rendering process to enhance the photorealistic insertion of virtual objects into real-world scenes.
diffusion models inverse rendering physically based rendering scene lighting tone-mapping materials refinement