title: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
summary: For Mixture-of-Experts (MoE) models, an unbalanced expert load will lead to
  routing collapse or increased computational overhead. Existing methods commonly
  employ an auxiliary loss to encourage load balance, but a large auxiliary loss
  will introduce non-negligible interference gradients into training and thus
  impair the model performance. In order to control load balance while not
  producing undesired gradients during training, we propose Loss-Free Balancing,
  featured by an auxiliary-loss-free load balancing strategy. To be specific,
  before the top-K routing decision, Loss-Free Balancing will first apply an
  expert-wise bias to the routing scores of each expert. By dynamically updating
  the bias of each expert according to its recent load, Loss-Free Balancing can
  consistently maintain a balanced distribution of expert load. In addition,
  since Loss-Free Balancing does not produce any interference gradients, it also
  elevates the upper bound of model performance gained from MoE training. We
  validate the performance of Loss-Free Balancing on MoE models with up to 3B
  parameters trained on up to 200B tokens. Experimental results show that
  Loss-Free Balancing achieves both better performance and better load balance
  compared with traditional auxiliary-loss-controlled load balancing strategies.
paper_authors._id: 66d0746981e0683bca3fb7b0
paper_authors.name: Lean Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-02T07:19:01.270Z
paper_authors._id: 650c509472afb1e60e6151ae
paper_authors.avatarUrl: /avatars/c16ab5053a586819dc2b965303215ff7.svg
paper_authors.fullname: Lean Wang
paper_authors.type: user
paper_authors.user: AdaHousman
paper_authors._id: 66d0746981e0683bca3fb7b1
paper_authors.name: Huazuo Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T16:47:01.712Z
paper_authors._id: 64e370be59aa5366642ac329
paper_authors.avatarUrl: /avatars/0fa1eb6ac6c1aeff3e65bc86a6617f64.svg
paper_authors.fullname: Huazuo Gao
paper_authors.type: user
paper_authors.user: gaohuazuo
paper_authors._id: 66d0746981e0683bca3fb7b2
paper_authors.name: Chenggang Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T16:47:07.557Z
paper_authors._id: 66053b1f9e3555d648b21c3d
paper_authors.avatarUrl: /avatars/c8b33e7f702c4edb17add47f0eafe5e6.svg
paper_authors.fullname: Chenggang Zhao
paper_authors.type: user
paper_authors.user: LyricZ
paper_authors._id: 66d0746981e0683bca3fb7b3
paper_authors.name: Xu Sun
paper_authors._id: 66d0746981e0683bca3fb7b4
paper_authors.name: Damai Dai
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-08-30T02:45:15.707Z
paper_authors._id: 659389f8de82e1ef7b9a8b13
paper_authors.avatarUrl: /avatars/896ed9f4cdbd317493b303d070b7e12a.svg
paper_authors.fullname: Damai Dai
paper_authors.type: user
paper_authors.user: DeepSeekDDM
paper_title: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
paper_summary: For Mixture-of-Experts (MoE) models, an unbalanced expert load will lead to
  routing collapse or increased computational overhead. Existing methods commonly
  employ an auxiliary loss to encourage load balance, but a large auxiliary loss
  will introduce non-negligible interference gradients into training and thus
  impair the model performance. In order to control load balance while not
  producing undesired gradients during training, we propose Loss-Free Balancing,
  featured by an auxiliary-loss-free load balancing strategy. To be specific,
  before the top-K routing decision, Loss-Free Balancing will first apply an
  expert-wise bias to the routing scores of each expert. By dynamically updating
  the bias of each expert according to its recent load, Loss-Free Balancing can
  consistently maintain a balanced distribution of expert load. In addition,
  since Loss-Free Balancing does not produce any interference gradients, it also
  elevates the upper bound of model performance gained from MoE training. We
  validate the performance of Loss-Free Balancing on MoE models with up to 3B
  parameters trained on up to 200B tokens. Experimental results show that
  Loss-Free Balancing achieves both better performance and better load balance
  compared with traditional auxiliary-loss-controlled load balancing strategies.
paper_ai_keywords: Mixture-of-Experts (MoE) models routing collapse computational overhead auxiliary loss Loss-Free Balancing top-K expert-wise bias routing scores dynamic updating expert load bounds of model performance precision bias