MeshCraft: Exploring Efficient and Controllable Mesh Generation with
  Flow-based DiTs
In the domain of 3D content creation, achieving optimal mesh topology through
AI models has long been a pursuit for 3D artists. Previous methods, such as
MeshGPT, have explored the generation of ready-to-use 3D objects via mesh
auto-regressive techniques. While these methods produce visually impressive
results, their reliance on token-by-token predictions in the auto-regressive
process leads to several significant limitations. These include extremely slow
generation speeds and an uncontrollable number of mesh faces. In this paper, we
introduce MeshCraft, a novel framework for efficient and controllable mesh
generation, which leverages continuous spatial diffusion to generate discrete
triangle faces. Specifically, MeshCraft consists of two core components: 1) a
transformer-based VAE that encodes raw meshes into continuous face-level tokens
and decodes them back to the original meshes, and 2) a flow-based diffusion
transformer conditioned on the number of faces, enabling the generation of
high-quality 3D meshes with a predefined number of faces. By utilizing the
diffusion model for the simultaneous generation of the entire mesh topology,
MeshCraft achieves high-fidelity mesh generation at significantly faster speeds
compared to auto-regressive methods. Specifically, MeshCraft can generate an
800-face mesh in just 3.2 seconds (35times faster than existing baselines).
Extensive experiments demonstrate that MeshCraft outperforms state-of-the-art
techniques in both qualitative and quantitative evaluations on ShapeNet dataset
and demonstrates superior performance on Objaverse dataset. Moreover, it
integrates seamlessly with existing conditional guidance strategies, showcasing
its potential to relieve artists from the time-consuming manual work involved
in mesh creation.
67ebadecf9f9390b4cd1c6d9
Xianglong He
67ebadecf9f9390b4cd1c6da
Junyi Chen
67ebadecf9f9390b4cd1c6db
Di Huang
67ebadecf9f9390b4cd1c6dc
Zexiang Liu
67ebadecf9f9390b4cd1c6dd
Xiaoshui Huang
67ebadecf9f9390b4cd1c6de
Wanli Ouyang
67ebadecf9f9390b4cd1c6df
Chun Yuan
67ebadecf9f9390b4cd1c6e0
Yangguang Li
MeshCraft: Exploring Efficient and Controllable Mesh Generation with
  Flow-based DiTs
In the domain of 3D content creation, achieving optimal mesh topology through
AI models has long been a pursuit for 3D artists. Previous methods, such as
MeshGPT, have explored the generation of ready-to-use 3D objects via mesh
auto-regressive techniques. While these methods produce visually impressive
results, their reliance on token-by-token predictions in the auto-regressive
process leads to several significant limitations. These include extremely slow
generation speeds and an uncontrollable number of mesh faces. In this paper, we
introduce MeshCraft, a novel framework for efficient and controllable mesh
generation, which leverages continuous spatial diffusion to generate discrete
triangle faces. Specifically, MeshCraft consists of two core components: 1) a
transformer-based VAE that encodes raw meshes into continuous face-level tokens
and decodes them back to the original meshes, and 2) a flow-based diffusion
transformer conditioned on the number of faces, enabling the generation of
high-quality 3D meshes with a predefined number of faces. By utilizing the
diffusion model for the simultaneous generation of the entire mesh topology,
MeshCraft achieves high-fidelity mesh generation at significantly faster speeds
compared to auto-regressive methods. Specifically, MeshCraft can generate an
800-face mesh in just 3.2 seconds (35times faster than existing baselines).
Extensive experiments demonstrate that MeshCraft outperforms state-of-the-art
techniques in both qualitative and quantitative evaluations on ShapeNet dataset
and demonstrates superior performance on Objaverse dataset. Moreover, it
integrates seamlessly with existing conditional guidance strategies, showcasing
its potential to relieve artists from the time-consuming manual work involved
in mesh creation.
MeshCraft uses continuous spatial diffusion and transformer-based VAE for efficient, high-fidelity, and controllable 3D mesh generation, outperforming auto-regressive methods.
mesh auto-regressive techniques continuous spatial diffusion transformer-based VAE flow-based diffusion transformer mesh topology ShapeNet Objaverse conditional guidance strategies