Neural LightRig: Unlocking Accurate Object Normal and Material
  Estimation with Multi-Light Diffusion
Recovering the geometry and materials of objects from a single image is
challenging due to its under-constrained nature. In this paper, we present
Neural LightRig, a novel framework that boosts intrinsic estimation by
leveraging auxiliary multi-lighting conditions from 2D diffusion priors.
Specifically, 1) we first leverage illumination priors from large-scale
diffusion models to build our multi-light diffusion model on a synthetic
relighting dataset with dedicated designs. This diffusion model generates
multiple consistent images, each illuminated by point light sources in
different directions. 2) By using these varied lighting images to reduce
estimation uncertainty, we train a large G-buffer model with a U-Net backbone
to accurately predict surface normals and materials. Extensive experiments
validate that our approach significantly outperforms state-of-the-art methods,
enabling accurate surface normal and PBR material estimation with vivid
relighting effects. Code and dataset are available on our project page at
https://projects.zxhezexin.com/neural-lightrig.
675b9a4634a23ade5100fde2
Zexin He
claimed_verified
2024-12-13T15:22:32.990Z
63a5d18a8fb23d08bb30094c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671811420431-noauth.png
Zexin He
user
zxhezexin
675b9a4634a23ade5100fde3
Tengfei Wang
675b9a4634a23ade5100fde4
Xin Huang
675b9a4634a23ade5100fde5
Xingang Pan
675b9a4634a23ade5100fde6
Ziwei Liu
Neural LightRig: Unlocking Accurate Object Normal and Material
  Estimation with Multi-Light Diffusion
Recovering the geometry and materials of objects from a single image is
challenging due to its under-constrained nature. In this paper, we present
Neural LightRig, a novel framework that boosts intrinsic estimation by
leveraging auxiliary multi-lighting conditions from 2D diffusion priors.
Specifically, 1) we first leverage illumination priors from large-scale
diffusion models to build our multi-light diffusion model on a synthetic
relighting dataset with dedicated designs. This diffusion model generates
multiple consistent images, each illuminated by point light sources in
different directions. 2) By using these varied lighting images to reduce
estimation uncertainty, we train a large G-buffer model with a U-Net backbone
to accurately predict surface normals and materials. Extensive experiments
validate that our approach significantly outperforms state-of-the-art methods,
enabling accurate surface normal and PBR material estimation with vivid
relighting effects. Code and dataset are available on our project page at
https://projects.zxhezexin.com/neural-lightrig.
Neural LightRig uses 2D diffusion priors and a U-Net backbone G-buffer model to accurately estimate geometry and materials from a single image under diverse lighting conditions.
diffusion priors diffusion models multi-light diffusion model synthetic relighting dataset U-Net backbone G-buffer model surface normals PBR material estimation vivid relighting effects