DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers
Diffusion models have demonstrated remarkable success in various image
generation tasks, but their performance is often limited by the uniform
processing of inputs across varying conditions and noise levels. To address
this limitation, we propose a novel approach that leverages the inherent
heterogeneity of the diffusion process. Our method, DiffMoE, introduces a
batch-level global token pool that enables experts to access global token
distributions during training, promoting specialized expert behavior. To
unleash the full potential of the diffusion process, DiffMoE incorporates a
capacity predictor that dynamically allocates computational resources based on
noise levels and sample complexity. Through comprehensive evaluation, DiffMoE
achieves state-of-the-art performance among diffusion models on ImageNet
benchmark, substantially outperforming both dense architectures with 3x
activated parameters and existing MoE approaches while maintaining 1x activated
parameters. The effectiveness of our approach extends beyond class-conditional
generation to more challenging tasks such as text-to-image generation,
demonstrating its broad applicability across different diffusion model
applications. Project Page: https://shiml20.github.io/DiffMoE/
67da83d1b05eff6d87a41f81
Minglei Shi
claimed_verified
2025-03-19T09:43:55.360Z
662887715d246621f33d2ce6
/avatars/3e0b1017b1e1bf284758ce840c174290.svg
Shi Minglei
user
MingleiShi
67da83d1b05eff6d87a41f82
Ziyang Yuan
admin_assigned
2025-03-21T12:54:09.791Z
66c15837b186e4f6a0dac80c
/avatars/a3b75d6945f1608e64a2fcff887a5024.svg
Ziyang Yuan
user
ziyangy
67da83d1b05eff6d87a41f83
Haotian Yang
admin_assigned
2025-03-21T12:54:19.207Z
64ba53dcbc787364968a7649
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KgbDlawW8n2s_V3kzhJLW.png
Haotian Yang
user
haotianxoxo
67da83d1b05eff6d87a41f84
Xintao Wang
claimed_verified
2025-03-21T12:46:06.302Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
67da83d1b05eff6d87a41f85
Mingwu Zheng
admin_assigned
2025-03-21T12:54:26.135Z
647dcab4becb41a272921250
/avatars/1e4fa540109c4963f54ccad9bd9e9b42.svg
Mingwu Zheng
user
MuonZ
67da83d1b05eff6d87a41f86
Xin Tao
67da83d1b05eff6d87a41f87
Wenliang Zhao
admin_assigned
2025-03-21T12:54:33.665Z
65be89df5e342a230dd04375
https://cdn-avatars.huggingface.co/v1/production/uploads/65be89df5e342a230dd04375/aoexyZRbHt2lwHE3iBrBv.jpeg
Wenliang Zhao
user
kennyzhao
67da83d1b05eff6d87a41f88
Wenzhao Zheng
67da83d1b05eff6d87a41f89
Jie Zhou
67da83d1b05eff6d87a41f8a
Jiwen Lu
admin_assigned
2025-03-21T12:55:09.982Z
66c44203ea476bea05e9fcd7
/avatars/b061eebec609446e669f5ad6365959f9.svg
lu
user
jiwenlu
67da83d1b05eff6d87a41f8b
Pengfei Wan
claimed_verified
2025-03-24T08:54:52.586Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
67da83d1b05eff6d87a41f8c
Di Zhang
67da83d1b05eff6d87a41f8d
Kun Gai
DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers
Diffusion models have demonstrated remarkable success in various image
generation tasks, but their performance is often limited by the uniform
processing of inputs across varying conditions and noise levels. To address
this limitation, we propose a novel approach that leverages the inherent
heterogeneity of the diffusion process. Our method, DiffMoE, introduces a
batch-level global token pool that enables experts to access global token
distributions during training, promoting specialized expert behavior. To
unleash the full potential of the diffusion process, DiffMoE incorporates a
capacity predictor that dynamically allocates computational resources based on
noise levels and sample complexity. Through comprehensive evaluation, DiffMoE
achieves state-of-the-art performance among diffusion models on ImageNet
benchmark, substantially outperforming both dense architectures with 3x
activated parameters and existing MoE approaches while maintaining 1x activated
parameters. The effectiveness of our approach extends beyond class-conditional
generation to more challenging tasks such as text-to-image generation,
demonstrating its broad applicability across different diffusion model
applications. Project Page: https://shiml20.github.io/DiffMoE/
DiffMoE, a novel diffusion model approach using a batch-level global token pool and capacity predictor, achieves state-of-the-art performance on ImageNet and excels in text-to-image generation with efficient parameter usage.
diffusion models global token pool experts capacity predictor ImageNet dense architectures MoE approaches class-conditional generation text-to-image generation