title: CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via
    Competition
summary: Sparse mixture of experts (SMoE) offers an appealing solution to scale up the
  model complexity beyond the mean of increasing the network's depth or width.
  However, we argue that effective SMoE training remains challenging because of
  the suboptimal routing process where experts that perform computation do not
  directly contribute to the routing process. In this work, we propose
  competition, a novel mechanism to route tokens to experts with the highest
  neural response. Theoretically, we show that the competition mechanism enjoys a
  better sample efficiency than the traditional softmax routing. Furthermore, we
  develop CompeteSMoE, a simple yet effective algorithm to train large language
  models by deploying a router to learn the competition policy, thus enjoying
  strong performances at a low training overhead. Our extensive empirical
  evaluations on both the visual instruction tuning and language pre-training
  tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE
  compared to state-of-the-art SMoE strategies. We have made the implementation
  available at: https://github.com/Fsoft-AIC/CompeteSMoE. This work is an
  improved version of the previous study at arXiv:2402.02526
paper_authors._id: 682d3787265177367e119f04
paper_authors.name: Nam V. Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:40:58.584Z
paper_authors._id: 64c2bea2ada7df214276913b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c2bea2ada7df214276913b/QFCtmCn439Afsr7uqyoMT.jpeg
paper_authors.fullname: Nguyen Van Nam
paper_authors.type: user
paper_authors.user: DavidNguyen
paper_authors._id: 682d3787265177367e119f05
paper_authors.name: Huy Nguyen
paper_authors._id: 682d3787265177367e119f06
paper_authors.name: Quang Pham
paper_authors._id: 682d3787265177367e119f07
paper_authors.name: Van Nguyen
paper_authors._id: 682d3787265177367e119f08
paper_authors.name: Savitha Ramasamy
paper_authors._id: 682d3787265177367e119f09
paper_authors.name: Nhat Ho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-21T10:26:55.401Z
paper_authors._id: 66216209da5d83251916f301
paper_authors.avatarUrl: /avatars/dd9d8977cbca95ae3309738b805474d4.svg
paper_authors.fullname: Nhat Ho
paper_authors.type: user
paper_authors.user: nhatho
paper_title: CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via
    Competition
paper_summary: Sparse mixture of experts (SMoE) offers an appealing solution to scale up the
  model complexity beyond the mean of increasing the network's depth or width.
  However, we argue that effective SMoE training remains challenging because of
  the suboptimal routing process where experts that perform computation do not
  directly contribute to the routing process. In this work, we propose
  competition, a novel mechanism to route tokens to experts with the highest
  neural response. Theoretically, we show that the competition mechanism enjoys a
  better sample efficiency than the traditional softmax routing. Furthermore, we
  develop CompeteSMoE, a simple yet effective algorithm to train large language
  models by deploying a router to learn the competition policy, thus enjoying
  strong performances at a low training overhead. Our extensive empirical
  evaluations on both the visual instruction tuning and language pre-training
  tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE
  compared to state-of-the-art SMoE strategies. We have made the implementation
  available at: https://github.com/Fsoft-AIC/CompeteSMoE. This work is an
  improved version of the previous study at arXiv:2402.02526
paper_ai_summary: CompeteSMoE enhances sparse mixture of experts (SMoE) by introducing a competition mechanism to improve routing efficiency and model performance in large language models.
paper_ai_keywords: sparse mixture of experts SMoE competition mechanism sample efficiency softmax routing CompeteSMoE visual instruction tuning language pre-training