title: DDT: Decoupled Diffusion Transformer
summary: Diffusion transformers have demonstrated remarkable generation quality,
  albeit requiring longer training iterations and numerous inference steps. In
  each denoising step, diffusion transformers encode the noisy inputs to extract
  the lower-frequency semantic component and then decode the higher frequency
  with identical modules. This scheme creates an inherent optimization dilemma:
  encoding low-frequency semantics necessitates reducing high-frequency
  components, creating tension between semantic encoding and high-frequency
  decoding. To resolve this challenge, we propose a new
  \color{ddtD}ecoupled \color{ddtD}iffusion
  \color{ddtT}ransformer~(\color{ddtDDT}), with a decoupled
  design of a dedicated condition encoder for semantic extraction alongside a
  specialized velocity decoder. Our experiments reveal that a more substantial
  encoder yields performance improvements as model size increases. For ImageNet
  256times256, Our DDT-XL/2 achieves a new state-of-the-art performance of
  {1.31 FID}~(nearly 4times faster training convergence compared to previous
  diffusion transformers). For ImageNet 512times512, Our DDT-XL/2 achieves a
  new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our
  decoupled architecture enhances inference speed by enabling the sharing
  self-condition between adjacent denoising steps. To minimize performance
  degradation, we propose a novel statistical dynamic programming approach to
  identify optimal sharing strategies.
paper_authors._id: 67f726dc0b5aa5777fd3a431
paper_authors.name: Shuai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:44:49.192Z
paper_authors._id: 66615c855fd9d736e670e0a9
paper_authors.avatarUrl: /avatars/0ff3127b513552432a7c651e21d7f283.svg
paper_authors.fullname: wangshuai
paper_authors.type: user
paper_authors.user: wangsssssss
paper_authors._id: 67f726dc0b5aa5777fd3a432
paper_authors.name: Zhi Tian
paper_authors._id: 67f726dc0b5aa5777fd3a433
paper_authors.name: Weilin Huang
paper_authors._id: 67f726dc0b5aa5777fd3a434
paper_authors.name: Limin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-10T07:58:42.903Z
paper_authors._id: 62c77f4352d8ae531f5511f9
paper_authors.avatarUrl: /avatars/50198ccb02ccd286975a4613fbabee28.svg
paper_authors.fullname: Limin Wang
paper_authors.type: user
paper_authors.user: lmwang
paper_title: DDT: Decoupled Diffusion Transformer
paper_summary: Diffusion transformers have demonstrated remarkable generation quality,
  albeit requiring longer training iterations and numerous inference steps. In
  each denoising step, diffusion transformers encode the noisy inputs to extract
  the lower-frequency semantic component and then decode the higher frequency
  with identical modules. This scheme creates an inherent optimization dilemma:
  encoding low-frequency semantics necessitates reducing high-frequency
  components, creating tension between semantic encoding and high-frequency
  decoding. To resolve this challenge, we propose a new
  \color{ddtD}ecoupled \color{ddtD}iffusion
  \color{ddtT}ransformer~(\color{ddtDDT}), with a decoupled
  design of a dedicated condition encoder for semantic extraction alongside a
  specialized velocity decoder. Our experiments reveal that a more substantial
  encoder yields performance improvements as model size increases. For ImageNet
  256times256, Our DDT-XL/2 achieves a new state-of-the-art performance of
  {1.31 FID}~(nearly 4times faster training convergence compared to previous
  diffusion transformers). For ImageNet 512times512, Our DDT-XL/2 achieves a
  new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our
  decoupled architecture enhances inference speed by enabling the sharing
  self-condition between adjacent denoising steps. To minimize performance
  degradation, we propose a novel statistical dynamic programming approach to
  identify optimal sharing strategies.
paper_ai_summary: A decoupled diffusion transformer improves performance and training speed in image generation by separating semantic extraction and high-frequency decoding.
paper_ai_keywords: diffusion transformers denoising steps decoupled design condition encoder velocity decoder semantic extraction high-frequency decoding DDT FID dynamic programming self-conditioning