23.1
40.3

15.1
50.1
0.0
26.9

15.9
33.6

12.8
40.8

10.0
42.3
1.2
25.1

25.9
33.0

12.6
44.5

11.0
41.4
17.7
35.5

15.1
63.0

4

Figure 2: Average zero performance of FLAN-MOE models versus FLAN-T5 dense models for
similar effective FLOPs per token over the 57 MMLU tasks and 23 BBH tasks.
Mixture-of-Experts