83.0e
91.0d
68.8e
50.5e
72.9d
84.9e
90.3e
46.4d

–
–

73.4e
40.9e
71.9e
51.4
68.0e
21.5e
38.0f

18

certain tasks, including small ones like CB, WSC, fail to improve. Finally, on closed book Trivia
QA, our model improves over the ﬁne-tuned baseline with SSM from Roberts et al. (2020), but fails
to produce gains over both GPT-3 and GLAM.
While not the focus of this paper, we present the quality differential between recent advances in
inference-only techniques like few-shot learning and ﬁne-tuning on these tasks (GPT-3 (Brown
et al., 2020), GLAM (Du et al., 2021) and Gopher (Rae et al., 2021)). As expected and observed
previously, ﬁne-tuning outperforms zero/one-shot learning, but has the disadvantage of requiring
additional training and different models for each task.

7 TRACING TOKENS THROUGH THE MODEL
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS