title: Neural LightRig: Unlocking Accurate Object Normal and Material
    Estimation with Multi-Light Diffusion
summary: Recovering the geometry and materials of objects from a single image is
  challenging due to its under-constrained nature. In this paper, we present
  Neural LightRig, a novel framework that boosts intrinsic estimation by
  leveraging auxiliary multi-lighting conditions from 2D diffusion priors.
  Specifically, 1) we first leverage illumination priors from large-scale
  diffusion models to build our multi-light diffusion model on a synthetic
  relighting dataset with dedicated designs. This diffusion model generates
  multiple consistent images, each illuminated by point light sources in
  different directions. 2) By using these varied lighting images to reduce
  estimation uncertainty, we train a large G-buffer model with a U-Net backbone
  to accurately predict surface normals and materials. Extensive experiments
  validate that our approach significantly outperforms state-of-the-art methods,
  enabling accurate surface normal and PBR material estimation with vivid
  relighting effects. Code and dataset are available on our project page at
  https://projects.zxhezexin.com/neural-lightrig.
paper_authors._id: 675b9a4634a23ade5100fde2
paper_authors.name: Zexin He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:32.990Z
paper_authors._id: 63a5d18a8fb23d08bb30094c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671811420431-noauth.png
paper_authors.fullname: Zexin He
paper_authors.type: user
paper_authors.user: zxhezexin
paper_authors._id: 675b9a4634a23ade5100fde3
paper_authors.name: Tengfei Wang
paper_authors._id: 675b9a4634a23ade5100fde4
paper_authors.name: Xin Huang
paper_authors._id: 675b9a4634a23ade5100fde5
paper_authors.name: Xingang Pan
paper_authors._id: 675b9a4634a23ade5100fde6
paper_authors.name: Ziwei Liu
paper_title: Neural LightRig: Unlocking Accurate Object Normal and Material
    Estimation with Multi-Light Diffusion
paper_summary: Recovering the geometry and materials of objects from a single image is
  challenging due to its under-constrained nature. In this paper, we present
  Neural LightRig, a novel framework that boosts intrinsic estimation by
  leveraging auxiliary multi-lighting conditions from 2D diffusion priors.
  Specifically, 1) we first leverage illumination priors from large-scale
  diffusion models to build our multi-light diffusion model on a synthetic
  relighting dataset with dedicated designs. This diffusion model generates
  multiple consistent images, each illuminated by point light sources in
  different directions. 2) By using these varied lighting images to reduce
  estimation uncertainty, we train a large G-buffer model with a U-Net backbone
  to accurately predict surface normals and materials. Extensive experiments
  validate that our approach significantly outperforms state-of-the-art methods,
  enabling accurate surface normal and PBR material estimation with vivid
  relighting effects. Code and dataset are available on our project page at
  https://projects.zxhezexin.com/neural-lightrig.
paper_ai_summary: Neural LightRig uses 2D diffusion priors and a U-Net backbone G-buffer model to accurately estimate geometry and materials from a single image under diverse lighting conditions.
paper_ai_keywords: diffusion priors diffusion models multi-light diffusion model synthetic relighting dataset U-Net backbone G-buffer model surface normals PBR material estimation vivid relighting effects