title: DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
summary: Recent Diffusion Transformers (e.g., DiT) have demonstrated their powerful
  effectiveness in generating high-quality 2D images. However, it is still being
  determined whether the Transformer architecture performs equally well in 3D
  shape generation, as previous 3D diffusion methods mostly adopted the U-Net
  architecture. To bridge this gap, we propose a novel Diffusion Transformer for
  3D shape generation, namely DiT-3D, which can directly operate the denoising
  process on voxelized point clouds using plain Transformers. Compared to
  existing U-Net approaches, our DiT-3D is more scalable in model size and
  produces much higher quality generations. Specifically, the DiT-3D adopts the
  design philosophy of DiT but modifies it by incorporating 3D positional and
  patch embeddings to adaptively aggregate input from voxelized point clouds. To
  reduce the computational cost of self-attention in 3D shape generation, we
  incorporate 3D window attention into Transformer blocks, as the increased 3D
  token length resulting from the additional dimension of voxels can lead to high
  computation. Finally, linear and devoxelization layers are used to predict the
  denoised point clouds. In addition, our transformer architecture supports
  efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on
  ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on
  the ShapeNet dataset demonstrate that the proposed DiT-3D achieves
  state-of-the-art performance in high-fidelity and diverse 3D point cloud
  generation. In particular, our DiT-3D decreases the 1-Nearest Neighbor Accuracy
  of the state-of-the-art method by 4.59 and increases the Coverage metric by
  3.51 when evaluated on Chamfer Distance.
paper_authors._id: 64a62fb898a3e01fd6929da0
paper_authors.name: Shentong Mo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-07T07:35:12.294Z
paper_authors._id: 631cbce0aa346997917f8faa
paper_authors.avatarUrl: /avatars/abb996361d170b341429b98b9c91dbda.svg
paper_authors.fullname: Shentong
paper_authors.type: user
paper_authors.user: stoneMo
paper_authors._id: 64a62fb898a3e01fd6929da1
paper_authors.name: Enze Xie
paper_authors._id: 64a62fb898a3e01fd6929da2
paper_authors.name: Ruihang Chu
paper_authors._id: 64a62fb898a3e01fd6929da3
paper_authors.name: Lewei Yao
paper_authors._id: 64a62fb898a3e01fd6929da4
paper_authors.name: Lanqing Hong
paper_authors._id: 64a62fb898a3e01fd6929da5
paper_authors.name: Matthias Nießner
paper_authors._id: 64a62fb898a3e01fd6929da6
paper_authors.name: Zhenguo Li
paper_title: DiT-3D: Exploring Plain Diffusion Transformers for 3D Shape Generation
paper_summary: Recent Diffusion Transformers (e.g., DiT) have demonstrated their powerful
  effectiveness in generating high-quality 2D images. However, it is still being
  determined whether the Transformer architecture performs equally well in 3D
  shape generation, as previous 3D diffusion methods mostly adopted the U-Net
  architecture. To bridge this gap, we propose a novel Diffusion Transformer for
  3D shape generation, namely DiT-3D, which can directly operate the denoising
  process on voxelized point clouds using plain Transformers. Compared to
  existing U-Net approaches, our DiT-3D is more scalable in model size and
  produces much higher quality generations. Specifically, the DiT-3D adopts the
  design philosophy of DiT but modifies it by incorporating 3D positional and
  patch embeddings to adaptively aggregate input from voxelized point clouds. To
  reduce the computational cost of self-attention in 3D shape generation, we
  incorporate 3D window attention into Transformer blocks, as the increased 3D
  token length resulting from the additional dimension of voxels can lead to high
  computation. Finally, linear and devoxelization layers are used to predict the
  denoised point clouds. In addition, our transformer architecture supports
  efficient fine-tuning from 2D to 3D, where the pre-trained DiT-2D checkpoint on
  ImageNet can significantly improve DiT-3D on ShapeNet. Experimental results on
  the ShapeNet dataset demonstrate that the proposed DiT-3D achieves
  state-of-the-art performance in high-fidelity and diverse 3D point cloud
  generation. In particular, our DiT-3D decreases the 1-Nearest Neighbor Accuracy
  of the state-of-the-art method by 4.59 and increases the Coverage metric by
  3.51 when evaluated on Chamfer Distance.
paper_ai_keywords: Diffusion Transformers DiT 3D shape generation voxelized point clouds 3D positional embeddings patch embeddings 3D window attention self-attention linear layers devoxelization layers efficient fine-tuning DiT-2D StateNet Chamfer Distance