DDT: Decoupled Diffusion Transformer
Diffusion transformers have demonstrated remarkable generation quality,
albeit requiring longer training iterations and numerous inference steps. In
each denoising step, diffusion transformers encode the noisy inputs to extract
the lower-frequency semantic component and then decode the higher frequency
with identical modules. This scheme creates an inherent optimization dilemma:
encoding low-frequency semantics necessitates reducing high-frequency
components, creating tension between semantic encoding and high-frequency
decoding. To resolve this challenge, we propose a new
\color{ddtD}ecoupled \color{ddtD}iffusion
\color{ddtT}ransformer~(\color{ddtDDT}), with a decoupled
design of a dedicated condition encoder for semantic extraction alongside a
specialized velocity decoder. Our experiments reveal that a more substantial
encoder yields performance improvements as model size increases. For ImageNet
256times256, Our DDT-XL/2 achieves a new state-of-the-art performance of
{1.31 FID}~(nearly 4times faster training convergence compared to previous
diffusion transformers). For ImageNet 512times512, Our DDT-XL/2 achieves a
new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our
decoupled architecture enhances inference speed by enabling the sharing
self-condition between adjacent denoising steps. To minimize performance
degradation, we propose a novel statistical dynamic programming approach to
identify optimal sharing strategies.
67f726dc0b5aa5777fd3a431
Shuai Wang
claimed_verified
2025-04-10T06:44:49.192Z
66615c855fd9d736e670e0a9
/avatars/0ff3127b513552432a7c651e21d7f283.svg
wangshuai
user
wangsssssss
67f726dc0b5aa5777fd3a432
Zhi Tian
67f726dc0b5aa5777fd3a433
Weilin Huang
67f726dc0b5aa5777fd3a434
Limin Wang
admin_assigned
2025-04-10T07:58:42.903Z
62c77f4352d8ae531f5511f9
/avatars/50198ccb02ccd286975a4613fbabee28.svg
Limin Wang
user
lmwang
DDT: Decoupled Diffusion Transformer
Diffusion transformers have demonstrated remarkable generation quality,
albeit requiring longer training iterations and numerous inference steps. In
each denoising step, diffusion transformers encode the noisy inputs to extract
the lower-frequency semantic component and then decode the higher frequency
with identical modules. This scheme creates an inherent optimization dilemma:
encoding low-frequency semantics necessitates reducing high-frequency
components, creating tension between semantic encoding and high-frequency
decoding. To resolve this challenge, we propose a new
\color{ddtD}ecoupled \color{ddtD}iffusion
\color{ddtT}ransformer~(\color{ddtDDT}), with a decoupled
design of a dedicated condition encoder for semantic extraction alongside a
specialized velocity decoder. Our experiments reveal that a more substantial
encoder yields performance improvements as model size increases. For ImageNet
256times256, Our DDT-XL/2 achieves a new state-of-the-art performance of
{1.31 FID}~(nearly 4times faster training convergence compared to previous
diffusion transformers). For ImageNet 512times512, Our DDT-XL/2 achieves a
new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our
decoupled architecture enhances inference speed by enabling the sharing
self-condition between adjacent denoising steps. To minimize performance
degradation, we propose a novel statistical dynamic programming approach to
identify optimal sharing strategies.
A decoupled diffusion transformer improves performance and training speed in image generation by separating semantic extraction and high-frequency decoding.
diffusion transformers denoising steps decoupled design condition encoder velocity decoder semantic extraction high-frequency decoding DDT FID dynamic programming self-conditioning