6.4
8.3 (+1.9)
14.8 (+8.4)
20.5 (+14.1)
21.9 (+15.5)
28.4
29.0 (+0.4)
33.4 (+6.0)
37.9 (+9.5)
38.8 (+10.4)
49.1
52.6 (+3.5)
57.0 (+7.9)
57.5 (+8.4)
58.5 (+9.4)

Table 3: Increasing the number of tasks in the ﬁnetuning data improves performance of Flan-PaLM on
most evaluation benchmarks. The benchmark suites are MMLU (57 tasks), BBH (23 tasks), TyDiQA (8
languages), and MGSM (10 languages). The evaluation metric on all four benchmark suites is few-shot
prompted accuracy (exact match), where we take an unweighted average over all tasks. As an aggregate
metric we report the normalized average of MMLU-direct, MMLU-CoT, BBH-direct, BBH-CoT, TyDiQA, and
MGSM. These evaluation benchmarks are held-out (not included in the ﬁnetuning data).

4 Finetuning with chain-of-thought annotations
Scaling Instruction-Finetuned Language Models