Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali
Farhadi, and Yejin Choi. 2019. HellaSwag: Can
In Pro-
a machine really ﬁnish your sentence?
ceedings of the 57th Annual Meeting of the Asso-
ciation for Computational Linguistics, pages 4791–
4800, Florence, Italy. Association for Computational
Linguistics.

Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng
Gao, Kevin Duh, and Benjamin Van Durme. 2018.
Record: Bridging the gap between human and ma-
chine commonsense reading comprehension. CoRR,
abs/1810.12885.

Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jia-
jun Liang. 2022. Decoupled knowledge distillation.
In IEEE/CVF Conference on Computer Vision and
Pattern Recognition, CVPR 2022, New Orleans, LA,
USA, June 18-24, 2022, pages 11943–11952. IEEE.

A Prompt with Topics
For the prompts with topics, besides three random
examples, we sample three random topics from
the common topic list and present an example in
Figure 7.

B Response Generation
LaMini-LM- A Diverse Herd of Distilled Models from Large-Scale Instructions