An important goal of artiﬁcial intelligence is to develop models that can generalize to unseen tasks. In natural
language processing (NLP), pretrained language models have made signiﬁcant progress towards this goal,
as they can perform tasks given natural language descriptions (Brown et al., 2020, inter alia). Further progress
has been made by ﬁnetuning language models on a collection of tasks phrased as instructions, which enables
models to respond better to instructions and reduces the need for few-shot exemplars (Ouyang et al., 2022;
Wei et al., 2021; Sanh et al., 2021, inter alia).
In this paper we advance instruction ﬁnetuning in several ways. First, we study the impact of scaling
on instruction ﬁnetuning. Our experiments show that instruction ﬁnetuning does scale well with the
number of tasks and the size of the model. Their respective scaling behaviors suggest that future research
Scaling Instruction-Finetuned Language Models