20

Table 10: BBH[9:18] individual task performance.

BBH

Logical Deduction

Logical Deduction

Movie

Seven Objects

Three Objects

Recommendation

Multistep
Arithmetic

Navigate

Object
Counting

Penguins
in a Table

Reasoning about
Colored Objects

Ruin
Names

Direct

Direct

Direct

Direct CoT Direct CoT Direct CoT Direct CoT Direct

Direct CoT

Model

-
-
-
-

davinci
text-davinci-002
text-davinci-003
code-davinci-002

80M T5-Small

Flan-T5-Small

250M T5-Base

Flan-T5-Base

780M T5-Large

Flan-T5-Large

3B

T5-XL
Flan-T5-XL

11B T5-XXL

Flan-T5-XXL

8B

PaLM
Flan-PaLM

62B PaLM

Flan-PaLM

540B PaLM

Flan-PaLM
250M SwitchBASE

11B SwitchXXL

FLAN-SwitchXXL

80M FLAN-GSSMALL
250M FLAN-GSBASE
780M FLAN-GSLARGE
80M FLAN-ECSMALL
250M FLAN-ECBASE
780M FLAN-ECLARGE
3B
250M STBASE

FLAN-ECXL

FLAN-STBASE

32B ST32B

FLAN-ST32B

20.0
26.8
40.0
26.0

13.2
16.8

14.8
24.4

13.2
46.8

13.6
53.6

18.0
54.8

13.2
25.6

19.6
48.8

24.8
50.8
0.0
38.4
Mixture-of-Experts