3 Experiment

We study FLAN-MOE in the context of instruction-tuning. We first perform a controlled comparison
of FLAN-MOE to an equivalent “standard” dense encoder-decoder Transformer (T5), across a range
of model sizes in Section 3.2. We subsequently demonstrate in Section 3.3 that scaling up our model,
referred to as FLAN-MOE, can attain remarkable performance levels. Our most extensive model,
FLAN-ST32B, surpasses the performance of FLAN-PALM62B while utilizing less than 30% of FLOPs
per token. We further ablate the various design decisions in the next Section.

3.1 Settings

Traning Data. By default, all models are trained on the 1,836 finetuning tasks by combining four
mixtures from prior work: Muffin, T0-SF, NIV2, and CoT, as in [4]. Specifically, Muffin comprises
80 tasks from [52] and 26 dialog/program synthesis tasks; T0-SF comprises 193 tasks from [44];
NIV2 comprises 1554 tasks from [51]; CoT comprises 9 reasoning tasks.
Mixture-of-Experts