Expert Race: A Flexible Routing Strategy for Scaling Diffusion
  Transformer with Mixture of Experts
Diffusion models have emerged as mainstream framework in visual generation.
Building upon this success, the integration of Mixture of Experts (MoE) methods
has shown promise in enhancing model scalability and performance. In this
paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with
a flexible routing strategy, Expert Race. By allowing tokens and experts to
compete together and select the top candidates, the model learns to dynamically
assign experts to critical tokens. Additionally, we propose per-layer
regularization to address challenges in shallow layer learning, and router
similarity loss to prevent mode collapse, ensuring better expert utilization.
Extensive experiments on ImageNet validate the effectiveness of our approach,
showcasing significant performance gains while promising scaling properties.
67dd04563b4c256a9809cc96
Yike Yuan
claimed_verified
2025-03-21T11:37:58.139Z
6656c30934f77bc39689a478
/avatars/226a797834d7353420ffe60cc93e8ee2.svg
Yike Yuan
user
yyk-wew
67dd04563b4c256a9809cc97
Ziyu Wang
67dd04563b4c256a9809cc98
Zihao Huang
claimed_verified
2025-03-21T11:38:05.513Z
65a62085576772f531e13856
/avatars/72c67a60422e333ea4e323f7480ae0b7.svg
Huang Zihao
user
FetchFortune
67dd04563b4c256a9809cc99
Defa Zhu
67dd04563b4c256a9809cc9a
Xun Zhou
67dd04563b4c256a9809cc9b
Jingyi Yu
67dd04563b4c256a9809cc9c
Qiyang Min
Expert Race: A Flexible Routing Strategy for Scaling Diffusion
  Transformer with Mixture of Experts
Diffusion models have emerged as mainstream framework in visual generation.
Building upon this success, the integration of Mixture of Experts (MoE) methods
has shown promise in enhancing model scalability and performance. In this
paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with
a flexible routing strategy, Expert Race. By allowing tokens and experts to
compete together and select the top candidates, the model learns to dynamically
assign experts to critical tokens. Additionally, we propose per-layer
regularization to address challenges in shallow layer learning, and router
similarity loss to prevent mode collapse, ensuring better expert utilization.
Extensive experiments on ImageNet validate the effectiveness of our approach,
showcasing significant performance gains while promising scaling properties.
Race-DiT, a novel MoE diffusion transformer, uses a flexible Expert Race routing strategy and regularization techniques to improve performance and scalability in image generation.
Mixture of Experts (MoE) diffusion transformers Expert Race per-layer regularization router similarity loss ImageNet