2.1 Finetuning Data
Task mixtures. Prior literature has shown that increasing the number of tasks in ﬁnetuning with instructions
improves generalization to unseen tasks (Wei et al., 2021; Sanh et al., 2021, inter alia). In this paper we scale
to 1,836 ﬁnetuning tasks by combining four mixtures from prior work: Muﬃn, T0-SF, NIV2, and CoT, as
summarized in Figure 2. Muﬃn3 (80 tasks) comprises 62 tasks from Wei et al. (2021) and 26 new tasks that
we added in this work, including dialog data (Byrne et al., 2019; Anantha et al., 2021; Dai et al., 2022) and
program synthesis data (Yasunaga and Liang, 2020; Li et al., 2022). T0-SF (193 tasks) comprises tasks from
T0 (Sanh et al., 2021) that do not overlap with the data used in Muﬃn (SF stands for “sans Flan”). NIV2
(1554 tasks) comprises tasks from Wang et al. (2022c).4
Scaling Instruction-Finetuned Language Models