title: Expert Race: A Flexible Routing Strategy for Scaling Diffusion
    Transformer with Mixture of Experts
summary: Diffusion models have emerged as mainstream framework in visual generation.
  Building upon this success, the integration of Mixture of Experts (MoE) methods
  has shown promise in enhancing model scalability and performance. In this
  paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with
  a flexible routing strategy, Expert Race. By allowing tokens and experts to
  compete together and select the top candidates, the model learns to dynamically
  assign experts to critical tokens. Additionally, we propose per-layer
  regularization to address challenges in shallow layer learning, and router
  similarity loss to prevent mode collapse, ensuring better expert utilization.
  Extensive experiments on ImageNet validate the effectiveness of our approach,
  showcasing significant performance gains while promising scaling properties.
paper_authors._id: 67dd04563b4c256a9809cc96
paper_authors.name: Yike Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:37:58.139Z
paper_authors._id: 6656c30934f77bc39689a478
paper_authors.avatarUrl: /avatars/226a797834d7353420ffe60cc93e8ee2.svg
paper_authors.fullname: Yike Yuan
paper_authors.type: user
paper_authors.user: yyk-wew
paper_authors._id: 67dd04563b4c256a9809cc97
paper_authors.name: Ziyu Wang
paper_authors._id: 67dd04563b4c256a9809cc98
paper_authors.name: Zihao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:38:05.513Z
paper_authors._id: 65a62085576772f531e13856
paper_authors.avatarUrl: /avatars/72c67a60422e333ea4e323f7480ae0b7.svg
paper_authors.fullname: Huang Zihao
paper_authors.type: user
paper_authors.user: FetchFortune
paper_authors._id: 67dd04563b4c256a9809cc99
paper_authors.name: Defa Zhu
paper_authors._id: 67dd04563b4c256a9809cc9a
paper_authors.name: Xun Zhou
paper_authors._id: 67dd04563b4c256a9809cc9b
paper_authors.name: Jingyi Yu
paper_authors._id: 67dd04563b4c256a9809cc9c
paper_authors.name: Qiyang Min
paper_title: Expert Race: A Flexible Routing Strategy for Scaling Diffusion
    Transformer with Mixture of Experts
paper_summary: Diffusion models have emerged as mainstream framework in visual generation.
  Building upon this success, the integration of Mixture of Experts (MoE) methods
  has shown promise in enhancing model scalability and performance. In this
  paper, we introduce Race-DiT, a novel MoE model for diffusion transformers with
  a flexible routing strategy, Expert Race. By allowing tokens and experts to
  compete together and select the top candidates, the model learns to dynamically
  assign experts to critical tokens. Additionally, we propose per-layer
  regularization to address challenges in shallow layer learning, and router
  similarity loss to prevent mode collapse, ensuring better expert utilization.
  Extensive experiments on ImageNet validate the effectiveness of our approach,
  showcasing significant performance gains while promising scaling properties.
paper_ai_summary: Race-DiT, a novel MoE diffusion transformer, uses a flexible Expert Race routing strategy and regularization techniques to improve performance and scalability in image generation.
paper_ai_keywords: Mixture of Experts (MoE) diffusion transformers Expert Race per-layer regularization router similarity loss ImageNet