MMLU
59.6
73.5
55.1

BBH
47.5
57.9
45.3

MMLU-CoT

BBH-CoT

56.9
70.9
48.6

44.9
66.3
41.4

Avg
49.9
67.2
47.6

55.7 (+1.1%)

45.9 (+1.3%)

52.2 (+7.4%)

42.7 (+3.1%)

49.1 (+3.2%)

Table 16: Comparisons (dev scores) of Flan models on CoT vs Direct.

We also tried some self-consistency (Wang et al., 2022b) experiments in combination with CoT. From brief
experiments, this raised CoT score from 53.9 to 57.1 (when the corresponding direct score was 55.4). This
shows that at 20B scale, CoT + self consistency can outperform direct prompting. We did not experiment
further since this increases the search space to a point where it was more time consuming than we would
have liked (or enjoyed). We leave any future experiments as an exercise for the reader.
UL2- Unifying Language Learning Paradigms