title: Dense-to-Sparse Gate for Mixture-of-Experts
abstract: Mixture-of-experts (MoE) is becoming popular due to its success in improving the model quality, especially in Transformers. By routing tokens with a sparse gate to a few experts that each only contains part of the full model, MoE keeps the model size unchanged and significantly reduces per-token computation, which effectively scales neural networks. However, we found that the current approach of jointly training experts and the sparse gate introduces a negative impact on model accuracy, diminishing the efficiency of expensive large-scale model training. In this work, we proposed $\texttt{Dense-To-Sparse}$ gate (DTS-Gate) for MoE training. Specifically, instead of using a permanent sparse gate, DTS-Gate begins as a dense gate that routes tokens to all experts, then gradually and adaptively becomes sparser while routes to fewer experts. MoE with DTS-Gate naturally decouples the training of experts and the sparse gate by training all experts at first and then learning the sparse gate.  Our code is available at https://anonymous.4open.science/r/MoE-3D0D/README.md/README.moe.md.
authors: Xiaonan Nie Shijie Cao Xupeng Miao Lingxiao Ma Jilong Xue Youshan Miao Zichao Yang Zhi Yang Bin CUI
hIndex: Zichao Yang 20