ZeroComp: Zero-shot Object Compositing from Image Intrinsics via
  Diffusion
We present ZeroComp, an effective zero-shot 3D object compositing approach
that does not require paired composite-scene images during training. Our method
leverages ControlNet to condition from intrinsic images and combines it with a
Stable Diffusion model to utilize its scene priors, together operating as an
effective rendering engine. During training, ZeroComp uses intrinsic images
based on geometry, albedo, and masked shading, all without the need for paired
images of scenes with and without composite objects. Once trained, it
seamlessly integrates virtual 3D objects into scenes, adjusting shading to
create realistic composites. We developed a high-quality evaluation dataset and
demonstrate that ZeroComp outperforms methods using explicit lighting
estimations and generative techniques in quantitative and human perception
benchmarks. Additionally, ZeroComp extends to real and outdoor image
compositing, even when trained solely on synthetic indoor data, showcasing its
effectiveness in image compositing.
670aa68d4bea0a5a627c6c23
Zitian Zhang
admin_assigned
2024-10-14T13:17:10.791Z
655dbda6668b64adf1495528
/avatars/dbf6072dcf0905873289d78209a92f87.svg
Zitian Zhang
user
zzt76
670aa68d4bea0a5a627c6c24
Frédéric Fortier-Chouinard
claimed_verified
2024-10-13T20:15:26.881Z
64763956fb22e3b77f3e13d6
/avatars/1e6acf5d733280931ba2ce099481b2be.svg
Frédéric Fortier-Chouinard
user
lefreud
670aa68d4bea0a5a627c6c25
Mathieu Garon
admin_assigned
2024-10-14T13:17:05.387Z
657df7d22bffc5568ae27a91
/avatars/a56fc5d1faaaff4b2b44f0d2c6e54ced.svg
Mathieu Garon
user
MathieuGaron
670aa68d4bea0a5a627c6c26
Anand Bhattad
admin_assigned
2024-10-14T13:16:30.640Z
6303d5924ec2dfa82a5627de
/avatars/1e433b7fb9d30b9f42dec487d658b1e3.svg
Anand Bhattad
user
anandbhattad
670aa68d4bea0a5a627c6c27
Jean-François Lalonde
admin_assigned
2024-10-14T13:16:36.336Z
66e9905f00d5cd69e36e0e1a
/avatars/082eb98a634ae2f46c3664c2e6d331a1.svg
Jean-Francois Lalonde
user
jflalonde
ZeroComp: Zero-shot Object Compositing from Image Intrinsics via
  Diffusion
We present ZeroComp, an effective zero-shot 3D object compositing approach
that does not require paired composite-scene images during training. Our method
leverages ControlNet to condition from intrinsic images and combines it with a
Stable Diffusion model to utilize its scene priors, together operating as an
effective rendering engine. During training, ZeroComp uses intrinsic images
based on geometry, albedo, and masked shading, all without the need for paired
images of scenes with and without composite objects. Once trained, it
seamlessly integrates virtual 3D objects into scenes, adjusting shading to
create realistic composites. We developed a high-quality evaluation dataset and
demonstrate that ZeroComp outperforms methods using explicit lighting
estimations and generative techniques in quantitative and human perception
benchmarks. Additionally, ZeroComp extends to real and outdoor image
compositing, even when trained solely on synthetic indoor data, showcasing its
effectiveness in image compositing.
ZeroComp combines ControlNet and Stable Diffusion to achieve zero-shot 3D object compositing without requiring paired training images, demonstrating superior performance in various benchmarks.
ControlNet Stable Diffusion intrinsic images geometry albedo masked shading rendering engine evaluation dataset explicit lighting estimations generative techniques indoor data real-world images outdoor image compositing