Conglong Li, Minjia Zhang, and Yuxiong He. Curriculum Learning: A Regularization Method for
Efﬁcient and Stable Billion-Scale GPT Model Pre-Training. arXiv:2108.06084 [cs], February
2022. URL http://arxiv.org/abs/2108.06084.

Frederick Liu, Siamak Shakeri, Hongkun Yu, and Jing Li. EncT5: Fine-tuning T5 Encoder for Non-
autoregressive Tasks. arxiv:2110.08426[cs], October 2021a. doi: 10.48550/arXiv.2110.08426.
URL http://arxiv.org/abs/2110.08426.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei
Han. On the Variance of the Adaptive Learning Rate and Beyond. In International Conference
on Learning Representations, March 2020a. URL https://openreview.net/forum?i
d=rkgz2aEKDr.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY