(which injects noise via dropout) was rarely unstable, we examined whether training noise might
improve the stability of sparse models. Table 3 shows a stability improvement versus the baseline,
but at the expense of lower quality. We also ﬁnd that input-jitter, introduced by Fedus et al. (2021),
diminishes quality at XL-scale, hence we ablate it in our models. Input-jitter multiplies the input
logits to the router by a uniform random variable between [1 − 10−2, 1 + 10−2] . Dropout in our
ablation is applied throughout the Transformer. As seen previously, improvements in small-scale
settings may fail to generalize when scaled up and therefore trends should always be monitored and
re-assessed at increasing scale (Kaplan et al., 2020).

Method
Baseline

Input jitter (10−2)

Dropout (0.1)

Fraction Stable

4/6
3/3
3/3

Quality (↑)
-1.755 ±0.02
-1.777 ±0.03
-1.822 ±0.11
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS