14

compute includes better architectures (So et al., 2021), improved training objectives (Tay et al., 2022a), and
better data (Du et al., 2022), among other work.

9 Conclusions

In this paper we have extended instruction ﬁnetuning and trained Flan-PaLM by (1) scaling to a 540B-
parameter language model, (2) scaling to 1.8K ﬁnetuning tasks, and (3) including chain-of-thought (CoT)
data in ﬁnetuning. Experiments show that model performance substantially improved with both larger
model size and more ﬁnetuning tasks. Moreover, whereas prior instruction ﬁnetuning methods degraded
performance on CoT tasks, jointly ﬁnetuning with CoT data improved performance on all evaluations.
Scaling Instruction-Finetuned Language Models