CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and
  Interactive Geometry Refiner
We present a novel generative 3D modeling system, coined CraftsMan, which can
generate high-fidelity 3D geometries with highly varied shapes, regular mesh
topologies, and detailed surfaces, and, notably, allows for refining the
geometry in an interactive manner. Despite the significant advancements in 3D
generation, existing methods still struggle with lengthy optimization
processes, irregular mesh topologies, noisy surfaces, and difficulties in
accommodating user edits, consequently impeding their widespread adoption and
implementation in 3D modeling software. Our work is inspired by the craftsman,
who usually roughs out the holistic figure of the work first and elaborates the
surface details subsequently. Specifically, we employ a 3D native diffusion
model, which operates on latent space learned from latent set-based 3D
representations, to generate coarse geometries with regular mesh topology in
seconds. In particular, this process takes as input a text prompt or a
reference image and leverages a powerful multi-view (MV) diffusion model to
generate multiple views of the coarse geometry, which are fed into our
MV-conditioned 3D diffusion model for generating the 3D geometry, significantly
improving robustness and generalizability. Following that, a normal-based
geometry refiner is used to significantly enhance the surface details. This
refinement can be performed automatically, or interactively with user-supplied
edits. Extensive experiments demonstrate that our method achieves high efficacy
in producing superior-quality 3D assets compared to existing methods. HomePage:
https://craftsman3d.github.io/, Code: https://github.com/wyysf-98/CraftsMan
6653e7c995bc4a27bafe985e
Weiyu Li
extracted_pending
2024-05-27T01:54:21.710Z
63b53090889aa6707f07afd6
/avatars/595ecfea802c6faed50e031d442f1639.svg
weiyuli
user
wyysf
6653e7c995bc4a27bafe985f
Jiarui Liu
6653e7c995bc4a27bafe9860
Rui Chen
extracted_pending
2024-05-27T01:54:21.710Z
650d3c195d57feb8471895f5
/avatars/826420dc293473d6518a251e62cf1e29.svg
Xuelin Chen
user
XuelinChen
6653e7c995bc4a27bafe9861
Yixun Liang
claimed_verified
2024-05-27T09:10:37.180Z
647d8f65becb41a272907e7a
/avatars/f891a1986a9361b23b7c83c23031cf26.svg
yixunliang
user
lyxun
6653e7c995bc4a27bafe9862
Xuelin Chen
admin_assigned
2024-05-27T08:41:02.590Z
650d3c195d57feb8471895f5
/avatars/826420dc293473d6518a251e62cf1e29.svg
Xuelin Chen
user
XuelinChen
6653e7c995bc4a27bafe9863
Ping Tan
6653e7c995bc4a27bafe9864
Xiaoxiao Long
CraftsMan: High-fidelity Mesh Generation with 3D Native Generation and
  Interactive Geometry Refiner
We present a novel generative 3D modeling system, coined CraftsMan, which can
generate high-fidelity 3D geometries with highly varied shapes, regular mesh
topologies, and detailed surfaces, and, notably, allows for refining the
geometry in an interactive manner. Despite the significant advancements in 3D
generation, existing methods still struggle with lengthy optimization
processes, irregular mesh topologies, noisy surfaces, and difficulties in
accommodating user edits, consequently impeding their widespread adoption and
implementation in 3D modeling software. Our work is inspired by the craftsman,
who usually roughs out the holistic figure of the work first and elaborates the
surface details subsequently. Specifically, we employ a 3D native diffusion
model, which operates on latent space learned from latent set-based 3D
representations, to generate coarse geometries with regular mesh topology in
seconds. In particular, this process takes as input a text prompt or a
reference image and leverages a powerful multi-view (MV) diffusion model to
generate multiple views of the coarse geometry, which are fed into our
MV-conditioned 3D diffusion model for generating the 3D geometry, significantly
improving robustness and generalizability. Following that, a normal-based
geometry refiner is used to significantly enhance the surface details. This
refinement can be performed automatically, or interactively with user-supplied
edits. Extensive experiments demonstrate that our method achieves high efficacy
in producing superior-quality 3D assets compared to existing methods. HomePage:
https://craftsman3d.github.io/, Code: https://github.com/wyysf-98/CraftsMan
CraftsMan uses a 3D native diffusion model for efficient generation and refinement of high-quality 3D models with regular mesh topologies and interactive user edits.
3D diffusion model latent space latent set-based 3D representations multi-view (MV) diffusion model MV-conditioned 3D diffusion model normal-based geometry refiner