Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun
Liu. TinyBERT: Distilling BERT for Natural Language Understanding. arXiv:1909.10351 [cs],
October 2020. URL http://arxiv.org/abs/1909.10351.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Span-
BERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the As-
sociation for Computational Linguistics, 8:64–77, 2020. doi: 10.1162/tacl a 00300. URL
https://aclanthology.org/2020.tacl-1.5.

Prabhu Kaliamoorthi, Aditya Siddhant, Edward Li, and Melvin Johnson. Distilling Large Language
Models into Tiny and Effective Students using pQRNN. arxiv: 2101.08890 [cs], January 2021.
URL https://arxiv.org/abs/2101.08890v1.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY