For simplicity and to cover dozens of tasks easily, we train on mixtures of the tasks listed rather than
separately ﬁne-tuning a model on each task. However, because the tasks vary in size considerably,
equally sampling per the number of examples would over-sample large tasks and under-sample small
ones. We therefore mix each task in proportion to the number of examples in its ‘train’ split (up to
some max num examples=65536) as in Raffel et al. (2019). This means that tasks containing
more than 65536 training examples are weighted as if they only contain max num examples.
Table 10 summarizes the quality of a dense T5-Large (L) model and sparse model with approxi-
mately the same number of FLOPs pre-trained for 500k steps with a 1M batch size (524B tokens)
on the C4 dataset (Raffel et al., 2019). The sequence length for the encoder was 512 and 114 for
the decoder. We observe improvements on the validation (dev) sets across a wide array of tasks ex-
ST-MOE- DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODELS