MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
This paper aims to generate materials for 3D meshes from text descriptions.
Unlike existing methods that synthesize texture maps, we propose to generate
segment-wise procedural material graphs as the appearance representation, which
supports high-quality rendering and provides substantial flexibility in
editing. Instead of relying on extensive paired data, i.e., 3D meshes with
material graphs and corresponding text descriptions, to train a material graph
generative model, we propose to leverage the pre-trained 2D diffusion model as
a bridge to connect the text and material graphs. Specifically, our approach
decomposes a shape into a set of segments and designs a segment-controlled
diffusion model to synthesize 2D images that are aligned with mesh parts. Based
on generated images, we initialize parameters of material graphs and fine-tune
them through the differentiable rendering module to produce materials in
accordance with the textual description. Extensive experiments demonstrate the
superior performance of our framework in photorealism, resolution, and
editability over existing methods. Project page:
https://zhanghe3z.github.io/MaPa/
662f06eb0f2654d615148212
Shangzhan Zhang
662f06eb0f2654d615148213
Sida Peng
admin_assigned
2024-04-29T12:07:05.676Z
62986ca2b58e71e2ac9b8f01
/avatars/83944db5f3dbb6f47c47c46fb2cb2849.svg
Sida Peng
user
pengsida
662f06eb0f2654d615148214
Tao Xu
662f06eb0f2654d615148215
Yuanbo Yang
admin_assigned
2024-04-29T12:08:12.891Z
65d5f8aa933e3da6e7d5d627
/avatars/b82ea295bf07f9060e781f07deebba31.svg
yangyuanbo
user
yangyuanbo
662f06eb0f2654d615148216
Tianrun Chen
662f06eb0f2654d615148217
Nan Xue
claimed_verified
2025-07-17T08:16:30.907Z
6485ce5ec7f19728a49df17a
/avatars/e83966e6906c1d0f151300981e30f85a.svg
Nan
user
cherubicxn
662f06eb0f2654d615148218
Yujun Shen
662f06eb0f2654d615148219
Hujun Bao
662f06eb0f2654d61514821a
Ruizhen Hu
662f06eb0f2654d61514821b
Xiaowei Zhou
MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
This paper aims to generate materials for 3D meshes from text descriptions.
Unlike existing methods that synthesize texture maps, we propose to generate
segment-wise procedural material graphs as the appearance representation, which
supports high-quality rendering and provides substantial flexibility in
editing. Instead of relying on extensive paired data, i.e., 3D meshes with
material graphs and corresponding text descriptions, to train a material graph
generative model, we propose to leverage the pre-trained 2D diffusion model as
a bridge to connect the text and material graphs. Specifically, our approach
decomposes a shape into a set of segments and designs a segment-controlled
diffusion model to synthesize 2D images that are aligned with mesh parts. Based
on generated images, we initialize parameters of material graphs and fine-tune
them through the differentiable rendering module to produce materials in
accordance with the textual description. Extensive experiments demonstrate the
superior performance of our framework in photorealism, resolution, and
editability over existing methods. Project page:
https://zhanghe3z.github.io/MaPa/
A method generates materials for 3D meshes from text by using segment-wise procedural material graphs and a diffusion model bridge.
diffusion model procedural material graphs segment-wise differentiable rendering photorealism resolution editability