call them T0 training and SUPERNI training for
short, respectively. To save the training budget, we
sampled 50K instances (but covering all their in-
structions)foreachdataset, whichhasacomparable
size to the instruction data we generated. Based on
the ﬁndings from Wang et al. (2022) and our early
experiments, reducing the number of instances per
task does not degrade the model’s generalization
performance to unseen tasks.
5.3 Experiment 1: Zero-Shot Generalization
We ﬁrst evaluate the models’ ability to follow in-
structions on typical NLP tasks in a zero-shot fash-
ion. We use the evaluation set of SUPERNI (Wang
etal.,2022),whichconsistsof119taskswith100in-
stances in each task. In this work, we mainly focus
on the zero-shot setup, i.e., the model is prompted
with the deﬁnition of the tasks only, without in-
context demonstration examples. For all our re-
quests to the GPT3 variants, we use the determin-
istic generation mode (temperature as 0 and no nu-
SELF-INSTRUCT- Aligning Language Model with Self Generated Instructions