n
o
e
g
a
r
e
v
a
d
e
z
i
l
a
m
r
o
N

)
%
(

s
k
s
a
t

t
u
o
-
d
l
e
h

60

40

20

0

n
o
e
g
a
r
e
v
a
d
e
z
i
l
a
m
r
o
N

)
%
(

s
k
s
a
t

t
u
o
-
d
l
e
h

60

40

20

0

0

1,836 tasks
282 tasks
89 tasks
9 tasks
No ﬁnetuning

540B model

62B model

8B model

8B

62B

540B

Model size (# parameters)

9

89

682 1,836
Number of ﬁnetuning tasks

282

Figure 4: Scaling behavior of multi-task instruction ﬁnetuning with respect to model size (# parameters) and
number of ﬁnetuning tasks. The x-axes are log scale. The benchmark suites are MMLU (57 tasks), BBH (23
tasks), TyDiQA (8 languages), and MGSM (10 languages). The evaluation metric on all four benchmark
suites is few-shot prompted accuracy (exact match), where we take an unweighted average over all tasks. As
an aggregate metric we report the normalized average of MMLU-direct, MMLU-CoT, BBH-direct, BBH-CoT,
TyDiQA, and MGSM. These evaluation benchmarks are held-out (not included in the ﬁnetuning data).

6

Tasks Norm. avg.

62B
Scaling Instruction-Finetuned Language Models