dynamic routing mixture of experts, 2021a.

Zhao You, Shulin Feng, Dan Su, and Dong Yu. Speechmoe2: Mixture-of-experts model with im-

Biao Zhang and Rico Sennrich.

Root mean square layer normalization.

arXiv preprint

proved routing, 2021b.

arXiv:1910.07467, 2019.

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. Pegasus: Pre-training with extracted

gap-sentences for abstractive summarization, 2020.

Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.
Record: Bridging the gap between human and machine commonsense reading comprehension.
arXiv preprint arXiv:1810.12885, 2018.

Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and

Jianfeng Gao. Taming sparsely activated transformer with stochastic experts, 2021.

30

A TOKEN LOAD BALANCE DESCRIPTION
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS