architecture for language models, given their po-
tential, as demonstrated in our experiments.
GPT-2 vs.
the
decoder-only models that we ﬁne-tune, we observe
a performance discrepancy among models that are
of comparable size. Based on the results in Table 4
and Figure 5, LaMini-GPT series is signiﬁcantly
superior on downstream NLP tasks compared to
LaMini-Cerebras, despite both having similar ar-
chitecture and size. Even more, LaMini-GPT of
LaMini-LM- A Diverse Herd of Distilled Models from Large-Scale Instructions