title: DiffiT: Diffusion Vision Transformers for Image Generation
summary: Diffusion models with their powerful expressivity and high sample quality
  have enabled many new applications and use-cases in various domains. For sample
  generation, these models rely on a denoising neural network that generates
  images by iterative denoising. Yet, the role of denoising network architecture
  is not well-studied with most efforts relying on convolutional residual U-Nets.
  In this paper, we study the effectiveness of vision transformers in
  diffusion-based generative learning. Specifically, we propose a new model,
  denoted as Diffusion Vision Transformers (DiffiT), which consists of a hybrid
  hierarchical architecture with a U-shaped encoder and decoder. We introduce a
  novel time-dependent self-attention module that allows attention layers to
  adapt their behavior at different stages of the denoising process in an
  efficient manner. We also introduce latent DiffiT which consists of transformer
  model with the proposed self-attention layers, for high-resolution image
  generation. Our results show that DiffiT is surprisingly effective in
  generating high-fidelity images, and it achieves state-of-the-art (SOTA)
  benchmarks on a variety of class-conditional and unconditional synthesis tasks.
  In the latent space, DiffiT achieves a new SOTA FID score of 1.73 on
  ImageNet-256 dataset. Repository: https://github.com/NVlabs/DiffiT
paper_authors._id: 656e8f3fb075b63c909d629f
paper_authors.name: Ali Hatamizadeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-06T10:25:38.486Z
paper_authors._id: 64414b62603214724ebd2636
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
paper_authors.fullname: Ali
paper_authors.type: user
paper_authors.user: ahatamiz
paper_authors._id: 656e8f3fb075b63c909d62a0
paper_authors.name: Jiaming Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T11:21:26.226Z
paper_authors._id: 6312c2f4bbaa385279d2da1b
paper_authors.avatarUrl: /avatars/2bf162889fa726ed18cc205b3f28609e.svg
paper_authors.fullname: Jiaming Song
paper_authors.type: user
paper_authors.user: jiamings
paper_authors._id: 656e8f3fb075b63c909d62a1
paper_authors.name: Guilin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:18:28.863Z
paper_authors._id: 63bc4115141c7d395c4aa02c
paper_authors.avatarUrl: /avatars/a9eca59e8c52cceab6524a4aaab11e38.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: liuguilin
paper_authors._id: 656e8f3fb075b63c909d62a2
paper_authors.name: Jan Kautz
paper_authors._id: 656e8f3fb075b63c909d62a3
paper_authors.name: Arash Vahdat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T11:22:08.057Z
paper_authors._id: 62f6956cd3bdacb7eec02920
paper_authors.avatarUrl: /avatars/b22db0823311f866c00db2efc4b9f814.svg
paper_authors.fullname: Arash Vahdat
paper_authors.type: user
paper_authors.user: avahdat
paper_title: DiffiT: Diffusion Vision Transformers for Image Generation
paper_summary: Diffusion models with their powerful expressivity and high sample quality
  have enabled many new applications and use-cases in various domains. For sample
  generation, these models rely on a denoising neural network that generates
  images by iterative denoising. Yet, the role of denoising network architecture
  is not well-studied with most efforts relying on convolutional residual U-Nets.
  In this paper, we study the effectiveness of vision transformers in
  diffusion-based generative learning. Specifically, we propose a new model,
  denoted as Diffusion Vision Transformers (DiffiT), which consists of a hybrid
  hierarchical architecture with a U-shaped encoder and decoder. We introduce a
  novel time-dependent self-attention module that allows attention layers to
  adapt their behavior at different stages of the denoising process in an
  efficient manner. We also introduce latent DiffiT which consists of transformer
  model with the proposed self-attention layers, for high-resolution image
  generation. Our results show that DiffiT is surprisingly effective in
  generating high-fidelity images, and it achieves state-of-the-art (SOTA)
  benchmarks on a variety of class-conditional and unconditional synthesis tasks.
  In the latent space, DiffiT achieves a new SOTA FID score of 1.73 on
  ImageNet-256 dataset. Repository: https://github.com/NVlabs/DiffiT
paper_ai_summary: A novel diffusion model using vision transformers with a hierarchical architecture and time-dependent self-attention achieves state-of-the-art performance in image generation.
paper_ai_keywords: diffusion models denoising neural network convolutional residual U-Nets vision transformers Diffusion Vision Transformers DiffiT U-shaped encoder decoder time-dependent self-attention latent DiffiT high-resolution image generation class-conditional synthesis unconditional synthesis FID score