are designed to be roughly FLOP matched variants of the T5-Large encoder-decoder models from
Raffel et al. (2019) with 770M parameters. The ST-MoE models have 32 experts with an expert
layer frequency of 1/4 (every fourth FFN layer is replaced by an MoE layer). The pre-training and
ﬁne-tuning train capacity factor is 1.25 and the eval is 2.0. We evaluate performance on the held-out
validation and train dataset partitions.
Across both tasks, the sparse model converges faster to 100% train set accuracy supporting that
sparse models optimize effectively under a data distribution shift. On the larger task, ReCORD, the
validation quality of the sparse model follows the boost in training and signiﬁcantly exceeds the
dense model. However, on the smaller task, CB, the sparse model lags its dense counterpart on held-
out data. As per the recommendation of Fedus et al. (2021), we consider increasing the dropout
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS