Biao Zhang and Rico Sennrich. Root Mean Square Layer Normalization. arXiv:1910.07467 [cs,

stat], October 2019. URL http://arxiv.org/abs/1910.07467.

Minjia Zhang and Yuxiong He. Accelerating Training of Transformer-Based Language Models with
Progressive Layer Dropping. In Advances in Neural Information Processing Systems, volume 33,
pp. 14011–14023, 2020. URL https://proceedings.neurips.cc/paper/2020/ha
sh/a1140a3d0df1c81e24ae954d935e8926-Abstract.html?fbclid=IwAR2b
OnieMB2yBSK6Ks9 RVAkPHfUnTQLliZzJEzp89cLiIvErF zf6efu6c.

Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W. Ronny Huang, and Tom Goldstein. GradInit:
Learning to Initialize Neural Networks for Stable and Efﬁcient Training. arxiv:2102.08098[cs],
November 2021. doi: 10.48550/arXiv.2102.08098. URL http://arxiv.org/abs/2102
.08098.

A OTHER MODIFICATIONS

APPENDIX
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY