Figure 2: An example of instruction generation prompt based on three random examples from self-instruct.

model once. Previous research has demonstrated
the effectiveness of sequence-level distillation. For
instance, Costa-jussà et al. (2022) used sequence-
level distillation to reduce the size of an NLLB
machine translation system to 600M parameters.
Similarly, by combining sequence-level distillation
with model pruning and quantization, Behnke et al.
(2021); Bogoychev et al. (2020) managed to train
a translation system that was approximately ×25
smaller than the teacher model without a signiﬁcant
decrease in BLEU score.
LaMini-LM- A Diverse Herd of Distilled Models from Large-Scale Instructions