Peter Izsak, Moshe Berchansky, and Omer Levy. How to Train BERT with an Academic Budget.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,
pp. 10644–10652, Online and Punta Cana, Dominican Republic, November 2021. Association
for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.831. URL https:
//aclanthology.org/2021.emnlp-main.831.

15

Preprint

Mojan Javaheripi, Shital Shah, Subhabrata Mukherjee, Tomasz L. Religa, Caio C. T. Mendes,
Gustavo H. de Rosa, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. LiteTrans-
formerSearch: Training-free On-device Search for Efﬁcient Autoregressive Language Models.
arXiv:2203.02094 [cs], March 2022. URL http://arxiv.org/abs/2203.02094.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic
Generalization Measures and Where to Find Them. arXiv:1912.02178 [cs, stat], December 2019.
URL http://arxiv.org/abs/1912.02178.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY