title: ControlMat: A Controlled Generative Approach to Material Capture
summary: Material reconstruction from a photograph is a key component of 3D content
  creation democratization. We propose to formulate this ill-posed problem as a
  controlled synthesis one, leveraging the recent progress in generative deep
  networks. We present ControlMat, a method which, given a single photograph with
  uncontrolled illumination as input, conditions a diffusion model to generate
  plausible, tileable, high-resolution physically-based digital materials. We
  carefully analyze the behavior of diffusion models for multi-channel outputs,
  adapt the sampling process to fuse multi-scale information and introduce rolled
  diffusion to enable both tileability and patched diffusion for high-resolution
  outputs. Our generative approach further permits exploration of a variety of
  materials which could correspond to the input image, mitigating the unknown
  lighting conditions. We show that our approach outperforms recent inference and
  latent-space-optimization methods, and carefully validate our diffusion process
  design choices. Supplemental materials and additional details are available at:
  https://gvecchio.com/controlmat/.
paper_authors._id: 64f82f091435cc7faa40db49
paper_authors.name: Giuseppe Vecchio
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-06T11:31:47.735Z
paper_authors._id: 643fa1c718afbc4d1f3e5e94
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643fa1c718afbc4d1f3e5e94/_cg3HHuycPGHnexJHbfwP.jpeg
paper_authors.fullname: Giuseppe Vecchio
paper_authors.type: user
paper_authors.user: gvecchio
paper_authors._id: 64f82f091435cc7faa40db4a
paper_authors.name: Rosalie Martin
paper_authors._id: 64f82f091435cc7faa40db4b
paper_authors.name: Arthur Roullier
paper_authors._id: 64f82f091435cc7faa40db4c
paper_authors.name: Adrien Kaiser
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-06T09:58:00.761Z
paper_authors._id: 63370068612e2c3c46d99434
paper_authors.avatarUrl: /avatars/cf91b83f058611046ce8e5e06a12cb13.svg
paper_authors.fullname: Adrien Kaiser
paper_authors.type: user
paper_authors.user: akaiser
paper_authors._id: 64f82f091435cc7faa40db4d
paper_authors.name: Romain Rouffet
paper_authors._id: 64f82f091435cc7faa40db4e
paper_authors.name: Valentin Deschaintre
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-06T08:35:32.438Z
paper_authors._id: 63e0dc30ab3fb81c39816ce7
paper_authors.avatarUrl: /avatars/a30c8ca8850ae120dd409b6f7d204493.svg
paper_authors.fullname: Valentin Deschaintre
paper_authors.type: user
paper_authors.user: deschain
paper_authors._id: 64f82f091435cc7faa40db4f
paper_authors.name: Tamy Boubekeur
paper_title: ControlMat: A Controlled Generative Approach to Material Capture
paper_summary: Material reconstruction from a photograph is a key component of 3D content
  creation democratization. We propose to formulate this ill-posed problem as a
  controlled synthesis one, leveraging the recent progress in generative deep
  networks. We present ControlMat, a method which, given a single photograph with
  uncontrolled illumination as input, conditions a diffusion model to generate
  plausible, tileable, high-resolution physically-based digital materials. We
  carefully analyze the behavior of diffusion models for multi-channel outputs,
  adapt the sampling process to fuse multi-scale information and introduce rolled
  diffusion to enable both tileability and patched diffusion for high-resolution
  outputs. Our generative approach further permits exploration of a variety of
  materials which could correspond to the input image, mitigating the unknown
  lighting conditions. We show that our approach outperforms recent inference and
  latent-space-optimization methods, and carefully validate our diffusion process
  design choices. Supplemental materials and additional details are available at:
  https://gvecchio.com/controlmat/.
paper_ai_summary: A method called ControlMat uses a diffusion model to generate high-resolution, tileable digital materials from a single photograph, addressing challenges of uncontrolled lighting.
paper_ai_keywords: diffusion model tileable high-resolution physically-based digital materials rolled diffusion patched diffusion multi-channel outputs multi-scale information generative approach latent-space-optimization methods