title: DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers
summary: Diffusion models have demonstrated remarkable success in various image
  generation tasks, but their performance is often limited by the uniform
  processing of inputs across varying conditions and noise levels. To address
  this limitation, we propose a novel approach that leverages the inherent
  heterogeneity of the diffusion process. Our method, DiffMoE, introduces a
  batch-level global token pool that enables experts to access global token
  distributions during training, promoting specialized expert behavior. To
  unleash the full potential of the diffusion process, DiffMoE incorporates a
  capacity predictor that dynamically allocates computational resources based on
  noise levels and sample complexity. Through comprehensive evaluation, DiffMoE
  achieves state-of-the-art performance among diffusion models on ImageNet
  benchmark, substantially outperforming both dense architectures with 3x
  activated parameters and existing MoE approaches while maintaining 1x activated
  parameters. The effectiveness of our approach extends beyond class-conditional
  generation to more challenging tasks such as text-to-image generation,
  demonstrating its broad applicability across different diffusion model
  applications. Project Page: https://shiml20.github.io/DiffMoE/
paper_authors._id: 67da83d1b05eff6d87a41f81
paper_authors.name: Minglei Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:43:55.360Z
paper_authors._id: 662887715d246621f33d2ce6
paper_authors.avatarUrl: /avatars/3e0b1017b1e1bf284758ce840c174290.svg
paper_authors.fullname: Shi Minglei
paper_authors.type: user
paper_authors.user: MingleiShi
paper_authors._id: 67da83d1b05eff6d87a41f82
paper_authors.name: Ziyang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:54:09.791Z
paper_authors._id: 66c15837b186e4f6a0dac80c
paper_authors.avatarUrl: /avatars/a3b75d6945f1608e64a2fcff887a5024.svg
paper_authors.fullname: Ziyang Yuan
paper_authors.type: user
paper_authors.user: ziyangy
paper_authors._id: 67da83d1b05eff6d87a41f83
paper_authors.name: Haotian Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:54:19.207Z
paper_authors._id: 64ba53dcbc787364968a7649
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KgbDlawW8n2s_V3kzhJLW.png
paper_authors.fullname: Haotian Yang
paper_authors.type: user
paper_authors.user: haotianxoxo
paper_authors._id: 67da83d1b05eff6d87a41f84
paper_authors.name: Xintao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T12:46:06.302Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 67da83d1b05eff6d87a41f85
paper_authors.name: Mingwu Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:54:26.135Z
paper_authors._id: 647dcab4becb41a272921250
paper_authors.avatarUrl: /avatars/1e4fa540109c4963f54ccad9bd9e9b42.svg
paper_authors.fullname: Mingwu Zheng
paper_authors.type: user
paper_authors.user: MuonZ
paper_authors._id: 67da83d1b05eff6d87a41f86
paper_authors.name: Xin Tao
paper_authors._id: 67da83d1b05eff6d87a41f87
paper_authors.name: Wenliang Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:54:33.665Z
paper_authors._id: 65be89df5e342a230dd04375
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65be89df5e342a230dd04375/aoexyZRbHt2lwHE3iBrBv.jpeg
paper_authors.fullname: Wenliang Zhao
paper_authors.type: user
paper_authors.user: kennyzhao
paper_authors._id: 67da83d1b05eff6d87a41f88
paper_authors.name: Wenzhao Zheng
paper_authors._id: 67da83d1b05eff6d87a41f89
paper_authors.name: Jie Zhou
paper_authors._id: 67da83d1b05eff6d87a41f8a
paper_authors.name: Jiwen Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:55:09.982Z
paper_authors._id: 66c44203ea476bea05e9fcd7
paper_authors.avatarUrl: /avatars/b061eebec609446e669f5ad6365959f9.svg
paper_authors.fullname: lu
paper_authors.type: user
paper_authors.user: jiwenlu
paper_authors._id: 67da83d1b05eff6d87a41f8b
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:54:52.586Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 67da83d1b05eff6d87a41f8c
paper_authors.name: Di Zhang
paper_authors._id: 67da83d1b05eff6d87a41f8d
paper_authors.name: Kun Gai
paper_title: DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers
paper_summary: Diffusion models have demonstrated remarkable success in various image
  generation tasks, but their performance is often limited by the uniform
  processing of inputs across varying conditions and noise levels. To address
  this limitation, we propose a novel approach that leverages the inherent
  heterogeneity of the diffusion process. Our method, DiffMoE, introduces a
  batch-level global token pool that enables experts to access global token
  distributions during training, promoting specialized expert behavior. To
  unleash the full potential of the diffusion process, DiffMoE incorporates a
  capacity predictor that dynamically allocates computational resources based on
  noise levels and sample complexity. Through comprehensive evaluation, DiffMoE
  achieves state-of-the-art performance among diffusion models on ImageNet
  benchmark, substantially outperforming both dense architectures with 3x
  activated parameters and existing MoE approaches while maintaining 1x activated
  parameters. The effectiveness of our approach extends beyond class-conditional
  generation to more challenging tasks such as text-to-image generation,
  demonstrating its broad applicability across different diffusion model
  applications. Project Page: https://shiml20.github.io/DiffMoE/
paper_ai_summary: DiffMoE, a novel diffusion model approach using a batch-level global token pool and capacity predictor, achieves state-of-the-art performance on ImageNet and excels in text-to-image generation with efficient parameter usage.
paper_ai_keywords: diffusion models global token pool experts capacity predictor ImageNet dense architectures MoE approaches class-conditional generation text-to-image generation