Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
<|reference_start|>Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts: By increasing model parameters but activating them sparsely when performing a task, the use of Mixture-of-Experts (MoE) architecture significantly improves the performance of Large Language Models (LLMs) without increasing the inference cost. However, the memory consumption due to the growing number of experts presents a challenge to the deployment of these models in many real world settings. Our empirical study reveals that some experts encode redundant knowledge during pre-training. We thus propose a method of grouping and pruning similar experts to improve the model's parameter efficiency. We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures, including Mixtral, Deepseek-MoE, and Qwen. The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks. We will release our code to facilitate future research.<|reference_end|>
@article{zhang2024diversifying,
    title={Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse
  Mixture-of-Experts},
    author={Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, Jianfeng Gao},
    journal={arXiv preprint arXiv:2407.09590},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.09590},
    primaryClass={cs.CL cs.LG}
}