title: MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
summary: This paper aims to generate materials for 3D meshes from text descriptions.
  Unlike existing methods that synthesize texture maps, we propose to generate
  segment-wise procedural material graphs as the appearance representation, which
  supports high-quality rendering and provides substantial flexibility in
  editing. Instead of relying on extensive paired data, i.e., 3D meshes with
  material graphs and corresponding text descriptions, to train a material graph
  generative model, we propose to leverage the pre-trained 2D diffusion model as
  a bridge to connect the text and material graphs. Specifically, our approach
  decomposes a shape into a set of segments and designs a segment-controlled
  diffusion model to synthesize 2D images that are aligned with mesh parts. Based
  on generated images, we initialize parameters of material graphs and fine-tune
  them through the differentiable rendering module to produce materials in
  accordance with the textual description. Extensive experiments demonstrate the
  superior performance of our framework in photorealism, resolution, and
  editability over existing methods. Project page:
  https://zhanghe3z.github.io/MaPa/
paper_authors._id: 662f06eb0f2654d615148212
paper_authors.name: Shangzhan Zhang
paper_authors._id: 662f06eb0f2654d615148213
paper_authors.name: Sida Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:07:05.676Z
paper_authors._id: 62986ca2b58e71e2ac9b8f01
paper_authors.avatarUrl: /avatars/83944db5f3dbb6f47c47c46fb2cb2849.svg
paper_authors.fullname: Sida Peng
paper_authors.type: user
paper_authors.user: pengsida
paper_authors._id: 662f06eb0f2654d615148214
paper_authors.name: Tao Xu
paper_authors._id: 662f06eb0f2654d615148215
paper_authors.name: Yuanbo Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:08:12.891Z
paper_authors._id: 65d5f8aa933e3da6e7d5d627
paper_authors.avatarUrl: /avatars/b82ea295bf07f9060e781f07deebba31.svg
paper_authors.fullname: yangyuanbo
paper_authors.type: user
paper_authors.user: yangyuanbo
paper_authors._id: 662f06eb0f2654d615148216
paper_authors.name: Tianrun Chen
paper_authors._id: 662f06eb0f2654d615148217
paper_authors.name: Nan Xue
paper_authors._id: 662f06eb0f2654d615148218
paper_authors.name: Yujun Shen
paper_authors._id: 662f06eb0f2654d615148219
paper_authors.name: Hujun Bao
paper_authors._id: 662f06eb0f2654d61514821a
paper_authors.name: Ruizhen Hu
paper_authors._id: 662f06eb0f2654d61514821b
paper_authors.name: Xiaowei Zhou
paper_title: MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
paper_summary: This paper aims to generate materials for 3D meshes from text descriptions.
  Unlike existing methods that synthesize texture maps, we propose to generate
  segment-wise procedural material graphs as the appearance representation, which
  supports high-quality rendering and provides substantial flexibility in
  editing. Instead of relying on extensive paired data, i.e., 3D meshes with
  material graphs and corresponding text descriptions, to train a material graph
  generative model, we propose to leverage the pre-trained 2D diffusion model as
  a bridge to connect the text and material graphs. Specifically, our approach
  decomposes a shape into a set of segments and designs a segment-controlled
  diffusion model to synthesize 2D images that are aligned with mesh parts. Based
  on generated images, we initialize parameters of material graphs and fine-tune
  them through the differentiable rendering module to produce materials in
  accordance with the textual description. Extensive experiments demonstrate the
  superior performance of our framework in photorealism, resolution, and
  editability over existing methods. Project page:
  https://zhanghe3z.github.io/MaPa/
paper_ai_summary: A method generates materials for 3D meshes from text by using segment-wise procedural material graphs and a diffusion model bridge.
paper_ai_keywords: diffusion model procedural material graphs segment-wise differentiable rendering photorealism resolution editability