We ﬁrst examine the eﬀect of scaling in terms of (1) the size of model and (2) the number of ﬁnetuning tasks
on performance on held-out tasks. We scale the model size by performing experiments on three PaLM model
sizes: 8B, 62B, and 540B. To scale the number of tasks, we sequentially add task mixtures starting from the
mixture with the fewest tasks to the mixture with the most tasks: CoT, Muﬃn, T0-SF, and NIV2.
Figure 4 shows the joint eﬀect of scaling these two variables on the normalized average of held-out benchmarks.
Individual benchmark results are reported in Table 3. First, we see that for all three model sizes shown,
multi-task instruction ﬁnetuning improves performance by a large margin compared to no ﬁnetuning. The
performance gain ranges from 9.4% to 15.5%.
Second, increasing the number of ﬁnetuning tasks improves performance, although the majority of the
improvement comes from using up to 282 tasks. There are two potential explanations for the small gain after
Scaling Instruction-Finetuned Language Models