MaterialFusion: Enhancing Inverse Rendering with Material Diffusion
  Priors
Recent works in inverse rendering have shown promise in using multi-view
images of an object to recover shape, albedo, and materials. However, the
recovered components often fail to render accurately under new lighting
conditions due to the intrinsic challenge of disentangling albedo and material
properties from input images. To address this challenge, we introduce
MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that
incorporates a 2D prior on texture and material properties. We present
StableMaterial, a 2D diffusion model prior that refines multi-lit data to
estimate the most likely albedo and material from given input appearances. This
model is trained on albedo, material, and relit image data derived from a
curated dataset of approximately ~12K artist-designed synthetic Blender objects
called BlenderVault. we incorporate this diffusion prior with an inverse
rendering framework where we use score distillation sampling (SDS) to guide the
optimization of the albedo and materials, improving relighting performance in
comparison with previous work. We validate MaterialFusion's relighting
performance on 4 datasets of synthetic and real objects under diverse
illumination conditions, showing our diffusion-aided approach significantly
improves the appearance of reconstructed objects under novel lighting
conditions. We intend to publicly release our BlenderVault dataset to support
further research in this field.
66f23ddf51dca07c005aff65
Yehonathan Litman
66f23ddf51dca07c005aff66
Or Patashnik
66f23ddf51dca07c005aff67
Kangle Deng
claimed_verified
2025-03-21T11:50:10.035Z
645d34ecce72244df7b29317
/avatars/1248933d9f89a15e67086325a8322d5e.svg
Kangle Deng
user
kangled
66f23ddf51dca07c005aff68
Aviral Agrawal
66f23ddf51dca07c005aff69
Rushikesh Zawar
66f23ddf51dca07c005aff6a
Fernando De la Torre
66f23ddf51dca07c005aff6b
Shubham Tulsiani
MaterialFusion: Enhancing Inverse Rendering with Material Diffusion
  Priors
Recent works in inverse rendering have shown promise in using multi-view
images of an object to recover shape, albedo, and materials. However, the
recovered components often fail to render accurately under new lighting
conditions due to the intrinsic challenge of disentangling albedo and material
properties from input images. To address this challenge, we introduce
MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that
incorporates a 2D prior on texture and material properties. We present
StableMaterial, a 2D diffusion model prior that refines multi-lit data to
estimate the most likely albedo and material from given input appearances. This
model is trained on albedo, material, and relit image data derived from a
curated dataset of approximately ~12K artist-designed synthetic Blender objects
called BlenderVault. we incorporate this diffusion prior with an inverse
rendering framework where we use score distillation sampling (SDS) to guide the
optimization of the albedo and materials, improving relighting performance in
comparison with previous work. We validate MaterialFusion's relighting
performance on 4 datasets of synthetic and real objects under diverse
illumination conditions, showing our diffusion-aided approach significantly
improves the appearance of reconstructed objects under novel lighting
conditions. We intend to publicly release our BlenderVault dataset to support
further research in this field.
The study introduces MaterialFusion, an inverse rendering pipeline that utilizes a 2D diffusion model, StableMaterial, to enhance the accuracy of albedo and material property estimation, thereby improving relightability of rendered objects under new lighting conditions.
inverse rendering MaterialFusion 2D diffusion model StableMaterial albedo material properties relighting performance score distillation sampling SDS BlenderVault