10 CONCLUSION

We temper the over-exuberance for scale in Fedus et al. (2021) by showing how a model with 1/5th
the size, but with a better balance of computation (FLOPs) to parameters – is a more effective sparse
learner. Furthermore, this improves the usability of sparse models since it can be deployed with less
memory overhead. Using our sparse model variant, we achieve SOTA across a wide range of the
most competitive public benchmarks. We hope this work shows the power of model sparsity and
accelerates the adoption of such models.

ACKNOWLEDGEMENTS

We would like to thank Alex Passos, Ekin Cubuk, Margaret Li, Noah Constant, Oriol Vinyals,
Basil Mustafa, Joan Puigcerver, Diego de Las Casas, Mike Lewis, and Ryan Sepassi for detailed
comments and feedback on early versions of the draft. We also thank the Google Brain Team for
useful discussions throughout the course of this work.

24
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS