title: MaterialFusion: Enhancing Inverse Rendering with Material Diffusion
    Priors
summary: Recent works in inverse rendering have shown promise in using multi-view
  images of an object to recover shape, albedo, and materials. However, the
  recovered components often fail to render accurately under new lighting
  conditions due to the intrinsic challenge of disentangling albedo and material
  properties from input images. To address this challenge, we introduce
  MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that
  incorporates a 2D prior on texture and material properties. We present
  StableMaterial, a 2D diffusion model prior that refines multi-lit data to
  estimate the most likely albedo and material from given input appearances. This
  model is trained on albedo, material, and relit image data derived from a
  curated dataset of approximately ~12K artist-designed synthetic Blender objects
  called BlenderVault. we incorporate this diffusion prior with an inverse
  rendering framework where we use score distillation sampling (SDS) to guide the
  optimization of the albedo and materials, improving relighting performance in
  comparison with previous work. We validate MaterialFusion's relighting
  performance on 4 datasets of synthetic and real objects under diverse
  illumination conditions, showing our diffusion-aided approach significantly
  improves the appearance of reconstructed objects under novel lighting
  conditions. We intend to publicly release our BlenderVault dataset to support
  further research in this field.
paper_authors._id: 66f23ddf51dca07c005aff65
paper_authors.name: Yehonathan Litman
paper_authors._id: 66f23ddf51dca07c005aff66
paper_authors.name: Or Patashnik
paper_authors._id: 66f23ddf51dca07c005aff67
paper_authors.name: Kangle Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:50:10.035Z
paper_authors._id: 645d34ecce72244df7b29317
paper_authors.avatarUrl: /avatars/1248933d9f89a15e67086325a8322d5e.svg
paper_authors.fullname: Kangle Deng
paper_authors.type: user
paper_authors.user: kangled
paper_authors._id: 66f23ddf51dca07c005aff68
paper_authors.name: Aviral Agrawal
paper_authors._id: 66f23ddf51dca07c005aff69
paper_authors.name: Rushikesh Zawar
paper_authors._id: 66f23ddf51dca07c005aff6a
paper_authors.name: Fernando De la Torre
paper_authors._id: 66f23ddf51dca07c005aff6b
paper_authors.name: Shubham Tulsiani
paper_title: MaterialFusion: Enhancing Inverse Rendering with Material Diffusion
    Priors
paper_summary: Recent works in inverse rendering have shown promise in using multi-view
  images of an object to recover shape, albedo, and materials. However, the
  recovered components often fail to render accurately under new lighting
  conditions due to the intrinsic challenge of disentangling albedo and material
  properties from input images. To address this challenge, we introduce
  MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that
  incorporates a 2D prior on texture and material properties. We present
  StableMaterial, a 2D diffusion model prior that refines multi-lit data to
  estimate the most likely albedo and material from given input appearances. This
  model is trained on albedo, material, and relit image data derived from a
  curated dataset of approximately ~12K artist-designed synthetic Blender objects
  called BlenderVault. we incorporate this diffusion prior with an inverse
  rendering framework where we use score distillation sampling (SDS) to guide the
  optimization of the albedo and materials, improving relighting performance in
  comparison with previous work. We validate MaterialFusion's relighting
  performance on 4 datasets of synthetic and real objects under diverse
  illumination conditions, showing our diffusion-aided approach significantly
  improves the appearance of reconstructed objects under novel lighting
  conditions. We intend to publicly release our BlenderVault dataset to support
  further research in this field.
paper_ai_summary: The study introduces MaterialFusion, an inverse rendering pipeline that utilizes a 2D diffusion model, StableMaterial, to enhance the accuracy of albedo and material property estimation, thereby improving relightability of rendered objects under new lighting conditions.
paper_ai_keywords: inverse rendering MaterialFusion 2D diffusion model StableMaterial albedo material properties relighting performance score distillation sampling SDS BlenderVault