Contributions

1. A large-scale study of the quality-stability trade-offs of stability techniques.
2. An introduction of the router z-loss that resolves instability issues, while slightly improv-

ing model quality.

3. A ﬁne-tuning analysis of sparse and dense models highlighting different hyperparameter
sensitivity to the batch size and learning rate. We show bad hyperparameters result in
virtually no ﬁne-tuning gain over dense models, despite large pre-training speed-ups.

4. Architectural, routing and model design principles for designing Pareto efﬁcient sparse

models in a distributed setting.

5. A qualitative analysis tracing token routing decisions across expert layers.
6. A 269B sparse model (the Stable Transferable Mixture-of-Experts or ST-MoE-32B)
which achieves state-of-the-art performance across a diverse set of natural language
benchmarks.

2 BACKGROUND
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS