63.6
81.8
90.9
72.7
9.1
9.1
18.2
45.5
18.2
36.4
27.3
54.5
36.4
81.8
18.2
54.5
72.7

Flan-U-PaLM

Flan-PaLM

16.7 25.9

90.9

0.0

7.4

8.3

540B PaLM

540B U-PaLM

3B

11B

8B

62B

62B

CoT
68.4
78.9
84.2
78.9
15.8
26.3
21.1
31.6
26.3
47.4
31.6
63.2
68.4
84.2
10.5
73.7
78.9
73.7
84.2
89.5
89.5
89.5
89.5
89.5

Direct CoT
39.7 40.5
63.1 60.0
64.8 64.6
68.2 64.5
26.7
5.6
28.7 12.1
25.7 14.5
35.9 33.7
25.1 15.0
45.1 40.5
25.7 14.5
52.4 45.5
25.9 18.7
55.1 48.6
24.3 24.1
49.3 41.3
55.1 49.0
59.6 56.9
61.2 57.6
66.1 62.0
71.3 62.9
73.5 70.9
71.5 64.0
74.1 69.8

40

D.2 BBH
BBH is a selection of challenging BIG-Bench tasks curated by Suzgun et al. (2022) for which model in
(Srivastava et al., 2022) achieved performance better than the average human rater. Suzgun et al. (2022) cites
23 tasks, with two tasks having three subtasks. For simplicity, we just count these subtasks as individual
tasks and take an unweighted average. We use the prompts given in Suzgun et al. (2022).
Scaling Instruction-Finetuned Language Models