Adaptive computation. Sparse models are a subclass of adaptive computation models since each
input gets different computation applied to it. In sparse models a token is routed to the expert(s)
of its choosing. When capacity factors are less than one, the model learns to not apply computa-
tion to certain tokens. This has shown promise in computer vision (Riquelme et al., 2021) and our
language experiments (Appendix D). We envision future models expanding this through heteroge-
neous experts (e.g. each expert applies differing computation). Intuitively, different input examples
will likely require different amounts of processing depending on difﬁculty. Future models in this
direction will be efﬁciently enabled through emerging computing infrastructures (Dean, 2021).
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS