Noam Shazeer and Mitchell Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory
Cost. arxiv:1804.04235[cs, stat], April 2018. doi: 10.48550/arXiv.1804.04235. URL http:
//arxiv.org/abs/1804.04235.

Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged Train-
ing for Transformer Language Models. arXiv:2203.06211 [cs], March 2022. URL http:
//arxiv.org/abs/2203.06211.

Sam Shleifer, Jason Weston, and Myle Ott. NormFormer: Improved Transformer Pretraining with
Extra Normalization. arXiv:2110.09456 [cs], November 2021. URL http://arxiv.org/
abs/2110.09456.

Leslie N. Smith and Nicholay Topin. Super-Convergence: Very Fast Training of Neural Networks
Using Large Learning Rates. arXiv:1708.07120 [cs, stat], May 2018. URL http://arxiv.
org/abs/1708.07120.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY