3.3 Scaling up FLAN-MOE

We increase the architecture size to assess the performance of FLAN-MOE in the large-scale regime.
As discussed above, we instruction fine-tune the largest ST-MoE32B [56] model with 12 expert layers
in encoder, and decoder, respectively; these are non-uniformly distributed, with 64 experts per layer,

6

50100150200Number of Steps (k)1020304050Avg. Held-Out ScoreMMLU-Direct Few-Shot50100150200Number of Steps (k)1520253035Avg. Held-Out ScoreBBH-Direct Few-Shot(a) Scaling (MMLU)

(b) Scaling (BBH)

(c) Routing (MMLU)

(d) Routing (BBH)

Figure 4: Average few-shot performance of FLAN-MOE models over the 57 MMLU tasks and 23
BBH tasks. (Different color represents different dense model sizes.)

Figure 5: Average few-shot performance of FLAN-MOE with different finetuning strategy.
Mixture-of-Experts