54.5 36.4 38.5
86.4 72.7 69.2
81.8 81.8 80.8
100.0 77.3 76.9

38.5
30.4 60.9 16.7 50.0 84.6
84.6
78.3 87.0 66.7 58.3 92.3
84.6
78.3 73.9 66.7 58.3 84.6
78.3 78.3 75.0 58.3 100.0 92.3

80M T5-Small

Flan-T5-Small

250M T5-Base

Flan-T5-Base

780M T5-Large

Flan-T5-Large

3B

T5-XL
Flan-T5-XL

11B T5-XXL

Flan-T5-XXL

8B

PaLM
Flan-PaLM

62B PaLM

Flan-PaLM

540B PaLM

Flan-PaLM
250M SwitchBASE

FLAN-SwitchBASE

11B SwitchXXL

FLAN-SwitchXXL

80M FLAN-GSSMALL
250M FLAN-GSBASE
780M FLAN-GSLARGE
80M FLAN-ECSMALL
250M FLAN-ECBASE
780M FLAN-ECLARGE
3B
250M STBASE

FLAN-ECXL

FLAN-STBASE

32B ST32B

FLAN-ST32B

0.0

40.9
30.8
50.0 31.8 15.4

0.0

18.2
30.8
59.1 50.0 50.0

0.0

13.6
30.8
54.5 54.5 57.7

0.0

18.2
30.8
72.7 72.7 57.7

0.0

22.7
34.6
63.6 63.6 73.1

36.4 31.8 15.4
72.7 54.5 61.5

77.3 40.9 57.7
81.8 54.5 80.8

90.9 72.7 88.5
90.9 95.5 88.5
27.3
11.5
50.0 36.4 46.2

0.0
Mixture-of-Experts