title: Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of
    Experts
summary: The Mixture of Experts (MoE) is an effective architecture for scaling large
  language models by leveraging sparse expert activation, optimizing the
  trade-off between performance and efficiency. However, under expert
  parallelism, MoE suffers from inference inefficiencies due to imbalanced
  token-to-expert assignment, where some experts are overloaded while others
  remain underutilized. This imbalance leads to poor resource utilization and
  increased latency, as the most burdened expert dictates the overall delay, a
  phenomenon we define as the \textit{Straggler Effect}. To mitigate
  this, we propose Capacity-Aware Inference, including two key techniques: (1)
  \textit{Capacity-Aware Token Drop}, which discards overloaded tokens
  to regulate the maximum latency of MoE, and (2) \textit{Capacity-Aware
  Token Reroute}, which reallocates overflowed tokens to underutilized experts,
  balancing the token distribution. These techniques collectively optimize both
  high-load and low-load expert utilization, leading to a more efficient MoE
  inference pipeline. Extensive experiments demonstrate the effectiveness of our
  methods, showing significant improvements in inference efficiency, e.g., 0.2\%
  average performance increase and a 1.94times inference speedup on
  Mixtral-8times7B-Instruct.
paper_authors._id: 67d1f73c843dbf5b7f980254
paper_authors.name: Shwai He
paper_authors._id: 67d1f73c843dbf5b7f980255
paper_authors.name: Weilin Cai
paper_authors._id: 67d1f73c843dbf5b7f980256
paper_authors.name: Jiayi Huang
paper_authors._id: 67d1f73c843dbf5b7f980257
paper_authors.name: Ang Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-20T02:26:15.788Z
paper_authors._id: 64a842b1221e6292c9fa7a45
paper_authors.avatarUrl: /avatars/7f67f357c830ed6dade7ed7548ac1f05.svg
paper_authors.fullname: Ang Li
paper_authors.type: user
paper_authors.user: charleslipku
paper_title: Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of
    Experts
paper_summary: The Mixture of Experts (MoE) is an effective architecture for scaling large
  language models by leveraging sparse expert activation, optimizing the
  trade-off between performance and efficiency. However, under expert
  parallelism, MoE suffers from inference inefficiencies due to imbalanced
  token-to-expert assignment, where some experts are overloaded while others
  remain underutilized. This imbalance leads to poor resource utilization and
  increased latency, as the most burdened expert dictates the overall delay, a
  phenomenon we define as the \textit{Straggler Effect}. To mitigate
  this, we propose Capacity-Aware Inference, including two key techniques: (1)
  \textit{Capacity-Aware Token Drop}, which discards overloaded tokens
  to regulate the maximum latency of MoE, and (2) \textit{Capacity-Aware
  Token Reroute}, which reallocates overflowed tokens to underutilized experts,
  balancing the token distribution. These techniques collectively optimize both
  high-load and low-load expert utilization, leading to a more efficient MoE
  inference pipeline. Extensive experiments demonstrate the effectiveness of our
  methods, showing significant improvements in inference efficiency, e.g., 0.2\%
  average performance increase and a 1.94times inference speedup on
  Mixtral-8times7B-Instruct.
paper_ai_summary: Capacity-Aware Inference techniques improve the efficiency of Mixture of Experts architecture by addressing the Straggler Effect through token drop and rerouting.
paper_ai_keywords: Mixture of Experts MoE sparse expert activation token-to-expert assignment Straggler Effect Capacity-Aware Inference Capacity-Aware Token Drop Capacity-Aware Token Reroute