4.1 HYPOTHESIS: A GENERALIZATION PROBLEM

Sparse models have performed remarkably well in the regime of large datasets, but have sometimes
performed poorly when ﬁne-tuning (Fedus et al., 2021; Artetxe et al., 2021). We present evidence
for a (not so surprising) hypothesis that sparse models are prone to overﬁtting. We illustrate this
problem through two tasks in SuperGLUE (Wang et al., 2019) – Commitment Bank (De Marneffe
et al., 2019) and ReCORD (Zhang et al., 2018). Commitment Bank (CB) has 250 training examples
while ReCORD has over 100,000. This signiﬁcant size discrepancy facilitates a natural study for
overﬁtting on two tasks selected as part of the same benchmark.
In Figure 3, we compare the ﬁne-tuning characteristics of the Dense L and the ST-MoE-L model.
Each model was pre-trained on 500B tokens from the C4 corpus (Raffel et al., 2019). The models
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS