Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
<|reference_start|>Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering: The correct insertion of virtual objects in images of real-world scenes requires a deep understanding of the scene's lighting, geometry and materials, as well as the image formation process. While recent large-scale diffusion models have shown strong generative and inpainting capabilities, we find that current models do not sufficiently "understand" the scene shown in a single picture to generate consistent lighting effects (shadows, bright reflections, etc.) while preserving the identity and details of the composited object. We propose using a personalized large diffusion model as guidance to a physically based inverse rendering process. Our method recovers scene lighting and tone-mapping parameters, allowing the photorealistic composition of arbitrary virtual objects in single frames or videos of indoor or outdoor scenes. Our physically based pipeline further enables automatic materials and tone-mapping refinement.<|reference_end|>
@article{liang2024photorealistic,
    title={Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering},
    author={Ruofan Liang, Zan Gojcic, Merlin Nimier-David, David Acuna, Nandita
  Vijaykumar, Sanja Fidler, Zian Wang},
    journal={arXiv preprint arXiv:2408.09702},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.09702},
    primaryClass={cs.CV cs.AI cs.GR}
}