CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via
  Competition
Sparse mixture of experts (SMoE) offers an appealing solution to scale up the
model complexity beyond the mean of increasing the network's depth or width.
However, we argue that effective SMoE training remains challenging because of
the suboptimal routing process where experts that perform computation do not
directly contribute to the routing process. In this work, we propose
competition, a novel mechanism to route tokens to experts with the highest
neural response. Theoretically, we show that the competition mechanism enjoys a
better sample efficiency than the traditional softmax routing. Furthermore, we
develop CompeteSMoE, a simple yet effective algorithm to train large language
models by deploying a router to learn the competition policy, thus enjoying
strong performances at a low training overhead. Our extensive empirical
evaluations on both the visual instruction tuning and language pre-training
tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE
compared to state-of-the-art SMoE strategies. We have made the implementation
available at: https://github.com/Fsoft-AIC/CompeteSMoE. This work is an
improved version of the previous study at arXiv:2402.02526
682d3787265177367e119f04
Nam V. Nguyen
claimed_verified
2025-05-21T08:40:58.584Z
64c2bea2ada7df214276913b
https://cdn-avatars.huggingface.co/v1/production/uploads/64c2bea2ada7df214276913b/QFCtmCn439Afsr7uqyoMT.jpeg
Nguyen Van Nam
user
DavidNguyen
682d3787265177367e119f05
Huy Nguyen
682d3787265177367e119f06
Quang Pham
682d3787265177367e119f07
Van Nguyen
682d3787265177367e119f08
Savitha Ramasamy
682d3787265177367e119f09
Nhat Ho
admin_assigned
2025-05-21T10:26:55.401Z
66216209da5d83251916f301
/avatars/dd9d8977cbca95ae3309738b805474d4.svg
Nhat Ho
user
nhatho
CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via
  Competition
Sparse mixture of experts (SMoE) offers an appealing solution to scale up the
model complexity beyond the mean of increasing the network's depth or width.
However, we argue that effective SMoE training remains challenging because of
the suboptimal routing process where experts that perform computation do not
directly contribute to the routing process. In this work, we propose
competition, a novel mechanism to route tokens to experts with the highest
neural response. Theoretically, we show that the competition mechanism enjoys a
better sample efficiency than the traditional softmax routing. Furthermore, we
develop CompeteSMoE, a simple yet effective algorithm to train large language
models by deploying a router to learn the competition policy, thus enjoying
strong performances at a low training overhead. Our extensive empirical
evaluations on both the visual instruction tuning and language pre-training
tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE
compared to state-of-the-art SMoE strategies. We have made the implementation
available at: https://github.com/Fsoft-AIC/CompeteSMoE. This work is an
improved version of the previous study at arXiv:2402.02526
CompeteSMoE enhances sparse mixture of experts (SMoE) by introducing a competition mechanism to improve routing efficiency and model performance in large language models.
sparse mixture of experts SMoE competition mechanism sample efficiency softmax routing CompeteSMoE visual instruction tuning language pre-training