DiffiT: Diffusion Vision Transformers for Image Generation
Diffusion models with their powerful expressivity and high sample quality
have enabled many new applications and use-cases in various domains. For sample
generation, these models rely on a denoising neural network that generates
images by iterative denoising. Yet, the role of denoising network architecture
is not well-studied with most efforts relying on convolutional residual U-Nets.
In this paper, we study the effectiveness of vision transformers in
diffusion-based generative learning. Specifically, we propose a new model,
denoted as Diffusion Vision Transformers (DiffiT), which consists of a hybrid
hierarchical architecture with a U-shaped encoder and decoder. We introduce a
novel time-dependent self-attention module that allows attention layers to
adapt their behavior at different stages of the denoising process in an
efficient manner. We also introduce latent DiffiT which consists of transformer
model with the proposed self-attention layers, for high-resolution image
generation. Our results show that DiffiT is surprisingly effective in
generating high-fidelity images, and it achieves state-of-the-art (SOTA)
benchmarks on a variety of class-conditional and unconditional synthesis tasks.
In the latent space, DiffiT achieves a new SOTA FID score of 1.73 on
ImageNet-256 dataset. Repository: https://github.com/NVlabs/DiffiT
656e8f3fb075b63c909d629f
Ali Hatamizadeh
claimed_verified
2023-12-06T10:25:38.486Z
64414b62603214724ebd2636
https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
Ali
user
ahatamiz
656e8f3fb075b63c909d62a0
Jiaming Song
admin_assigned
2023-12-05T11:21:26.226Z
6312c2f4bbaa385279d2da1b
/avatars/2bf162889fa726ed18cc205b3f28609e.svg
Jiaming Song
user
jiamings
656e8f3fb075b63c909d62a1
Guilin Liu
claimed_verified
2025-04-22T10:18:28.863Z
63bc4115141c7d395c4aa02c
/avatars/a9eca59e8c52cceab6524a4aaab11e38.svg
Liu
user
liuguilin
656e8f3fb075b63c909d62a2
Jan Kautz
656e8f3fb075b63c909d62a3
Arash Vahdat
admin_assigned
2023-12-05T11:22:08.057Z
62f6956cd3bdacb7eec02920
/avatars/b22db0823311f866c00db2efc4b9f814.svg
Arash Vahdat
user
avahdat
DiffiT: Diffusion Vision Transformers for Image Generation
Diffusion models with their powerful expressivity and high sample quality
have enabled many new applications and use-cases in various domains. For sample
generation, these models rely on a denoising neural network that generates
images by iterative denoising. Yet, the role of denoising network architecture
is not well-studied with most efforts relying on convolutional residual U-Nets.
In this paper, we study the effectiveness of vision transformers in
diffusion-based generative learning. Specifically, we propose a new model,
denoted as Diffusion Vision Transformers (DiffiT), which consists of a hybrid
hierarchical architecture with a U-shaped encoder and decoder. We introduce a
novel time-dependent self-attention module that allows attention layers to
adapt their behavior at different stages of the denoising process in an
efficient manner. We also introduce latent DiffiT which consists of transformer
model with the proposed self-attention layers, for high-resolution image
generation. Our results show that DiffiT is surprisingly effective in
generating high-fidelity images, and it achieves state-of-the-art (SOTA)
benchmarks on a variety of class-conditional and unconditional synthesis tasks.
In the latent space, DiffiT achieves a new SOTA FID score of 1.73 on
ImageNet-256 dataset. Repository: https://github.com/NVlabs/DiffiT
A novel diffusion model using vision transformers with a hierarchical architecture and time-dependent self-attention achieves state-of-the-art performance in image generation.
diffusion models denoising neural network convolutional residual U-Nets vision transformers Diffusion Vision Transformers DiffiT U-shaped encoder decoder time-dependent self-attention latent DiffiT high-resolution image generation class-conditional synthesis unconditional synthesis FID score