DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
<|reference_start|>DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models: Transformer-based Mixture-of-Experts (MoE) models have been driving several recent technological advancements in Natural Language Processing (NLP). These MoE models adopt a router mechanism to determine which experts to activate for routing input tokens. However, existing router mechanisms allocate a fixed number of experts to each token, which neglects the varying importance of different input tokens. In this study, we propose a novel dynamic router mechanism that Dynamically Allocates a variable number of experts for Mixture-of-Experts (DA-MoE) models based on an effective token importance measure. First, we show that the Transformer attention mechanism provides a natural and effective way of calculating token importance. Second, we propose a dynamic router mechanism that effectively decides the optimal number of experts (K) and allocates the top-K experts for each input token. Third, comprehensive experiments on several benchmark datasets demonstrate that our DA-MoE approach consistently outperforms the state-of-the-art Transformer based MoE model on the popular GLUE benchmark.<|reference_end|>
@article{aghdam2024da-moe:,
    title={DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models},
    author={Maryam Akhavan Aghdam, Hongpeng Jin, Yanzhao Wu},
    journal={arXiv preprint arXiv:2409.06669},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.06669},
    primaryClass={cs.LG}
}