title: Fast Training of Diffusion Transformer with Extreme Masking for 3D Point
    Clouds Generation
summary: Diffusion Transformers have recently shown remarkable effectiveness in
  generating high-quality 3D point clouds. However, training voxel-based
  diffusion models for high-resolution 3D voxels remains prohibitively expensive
  due to the cubic complexity of attention operators, which arises from the
  additional dimension of voxels. Motivated by the inherent redundancy of 3D
  compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer
  tailored for efficient 3D point cloud generation, which greatly reduces
  training costs. Specifically, we draw inspiration from masked autoencoders to
  dynamically operate the denoising process on masked voxelized point clouds. We
  also propose a novel voxel-aware masking strategy to adaptively aggregate
  background/foreground information from voxelized point clouds. Our method
  achieves state-of-the-art performance with an extreme masking ratio of nearly
  99%. Moreover, to improve multi-category 3D generation, we introduce
  Mixture-of-Expert (MoE) in 3D diffusion model. Each category can learn a
  distinct diffusion path with different experts, relieving gradient conflict.
  Experimental results on the ShapeNet dataset demonstrate that our method
  achieves state-of-the-art high-fidelity and diverse 3D point cloud generation
  performance. Our FastDiT-3D improves 1-Nearest Neighbor Accuracy and Coverage
  metrics when generating 128-resolution voxel point clouds, using only 6.5% of
  the original training cost.
paper_authors._id: 6579271b9999746238c9026c
paper_authors.name: Shentong Mo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:30:00.174Z
paper_authors._id: 631cbce0aa346997917f8faa
paper_authors.avatarUrl: /avatars/abb996361d170b341429b98b9c91dbda.svg
paper_authors.fullname: Shentong
paper_authors.type: user
paper_authors.user: stoneMo
paper_authors._id: 6579271b9999746238c9026d
paper_authors.name: Enze Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:30:36.057Z
paper_authors._id: 616961bdd3f656f79ad18ec1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616961bdd3f656f79ad18ec1/NYpxas378lDevRuUDepJJ.jpeg
paper_authors.fullname: xieenze
paper_authors.type: user
paper_authors.user: xieenze
paper_authors._id: 6579271b9999746238c9026e
paper_authors.name: Yue Wu
paper_authors._id: 6579271b9999746238c9026f
paper_authors.name: Junsong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-13T10:10:49.658Z
paper_authors._id: 645b5b09bc7518912e1f9733
paper_authors.avatarUrl: /avatars/4d35f728b41f93881a9b67c337f4d1df.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Lawrence-cj
paper_authors._id: 6579271b9999746238c90270
paper_authors.name: Matthias Nießner
paper_authors._id: 6579271b9999746238c90271
paper_authors.name: Zhenguo Li
paper_title: Fast Training of Diffusion Transformer with Extreme Masking for 3D Point
    Clouds Generation
paper_summary: Diffusion Transformers have recently shown remarkable effectiveness in
  generating high-quality 3D point clouds. However, training voxel-based
  diffusion models for high-resolution 3D voxels remains prohibitively expensive
  due to the cubic complexity of attention operators, which arises from the
  additional dimension of voxels. Motivated by the inherent redundancy of 3D
  compared to 2D, we propose FastDiT-3D, a novel masked diffusion transformer
  tailored for efficient 3D point cloud generation, which greatly reduces
  training costs. Specifically, we draw inspiration from masked autoencoders to
  dynamically operate the denoising process on masked voxelized point clouds. We
  also propose a novel voxel-aware masking strategy to adaptively aggregate
  background/foreground information from voxelized point clouds. Our method
  achieves state-of-the-art performance with an extreme masking ratio of nearly
  99%. Moreover, to improve multi-category 3D generation, we introduce
  Mixture-of-Expert (MoE) in 3D diffusion model. Each category can learn a
  distinct diffusion path with different experts, relieving gradient conflict.
  Experimental results on the ShapeNet dataset demonstrate that our method
  achieves state-of-the-art high-fidelity and diverse 3D point cloud generation
  performance. Our FastDiT-3D improves 1-Nearest Neighbor Accuracy and Coverage
  metrics when generating 128-resolution voxel point clouds, using only 6.5% of
  the original training cost.
paper_ai_keywords: Diffusion Transformers 3D point clouds voxel-based diffusion models attention operators masked diffusion transformer masked autoencoders denoising process voxelized point clouds voxel-aware masking strategy Mixture-of-Expert (MoE) ShapeNet dataset 1-Nearest Neighbor Accuracy Coverage metrics voxel point clouds