title: Scaling Diffusion Transformers to 16 Billion Parameters
summary: In this paper, we present DiT-MoE, a sparse version of the diffusion
  Transformer, that is scalable and competitive with dense networks while
  exhibiting highly optimized inference. The DiT-MoE includes two simple designs:
  shared expert routing and expert-level balance loss, thereby capturing common
  knowledge and reducing redundancy among the different routed experts. When
  applied to conditional image generation, a deep analysis of experts
  specialization gains some interesting observations: (i) Expert selection shows
  preference with spatial position and denoising time step, while insensitive
  with different class-conditional information; (ii) As the MoE layers go deeper,
  the selection of experts gradually shifts from specific spacial position to
  dispersion and balance. (iii) Expert specialization tends to be more
  concentrated at the early time step and then gradually uniform after half. We
  attribute it to the diffusion process that first models the low-frequency
  spatial information and then high-frequency complex information. Based on the
  above guidance, a series of DiT-MoE experimentally achieves performance on par
  with dense networks yet requires much less computational load during inference.
  More encouragingly, we demonstrate the potential of DiT-MoE with synthesized
  image data, scaling diffusion model at a 16.5B parameter that attains a new
  SoTA FID-50K score of 1.80 in 512times512 resolution settings. The project
  page: https://github.com/feizc/DiT-MoE.
paper_authors._id: 66973a469ad36faed155046e
paper_authors.name: Zhengcong Fei
paper_authors._id: 66973a469ad36faed155046f
paper_authors.name: Mingyuan Fan
paper_authors._id: 66973a469ad36faed1550470
paper_authors.name: Changqian Yu
paper_authors._id: 66973a469ad36faed1550471
paper_authors.name: Debang Li
paper_authors._id: 66973a469ad36faed1550472
paper_authors.name: Junshi Huang
paper_title: Scaling Diffusion Transformers to 16 Billion Parameters
paper_summary: In this paper, we present DiT-MoE, a sparse version of the diffusion
  Transformer, that is scalable and competitive with dense networks while
  exhibiting highly optimized inference. The DiT-MoE includes two simple designs:
  shared expert routing and expert-level balance loss, thereby capturing common
  knowledge and reducing redundancy among the different routed experts. When
  applied to conditional image generation, a deep analysis of experts
  specialization gains some interesting observations: (i) Expert selection shows
  preference with spatial position and denoising time step, while insensitive
  with different class-conditional information; (ii) As the MoE layers go deeper,
  the selection of experts gradually shifts from specific spacial position to
  dispersion and balance. (iii) Expert specialization tends to be more
  concentrated at the early time step and then gradually uniform after half. We
  attribute it to the diffusion process that first models the low-frequency
  spatial information and then high-frequency complex information. Based on the
  above guidance, a series of DiT-MoE experimentally achieves performance on par
  with dense networks yet requires much less computational load during inference.
  More encouragingly, we demonstrate the potential of DiT-MoE with synthesized
  image data, scaling diffusion model at a 16.5B parameter that attains a new
  SoTA FID-50K score of 1.80 in 512times512 resolution settings. The project
  page: https://github.com/feizc/DiT-MoE.
paper_ai_summary: DiT-MoE, a sparse diffusion Transformer with shared expert routing and expert-level balance loss, achieves competitive performance with dense networks in image generation while reducing computational load during inference.
paper_ai_keywords: diffusion Transformer sparse version shared expert routing expert-level balance loss conditional image generation expert specialization spatial position denoising time step class-conditional information diffusion process low-frequency spatial information high-frequency complex information synthesized image data FID-50K score