ISBN 978-1-4503-7998-4. doi: 10.1145/3394486.3406703. URL

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Min-
jia Zhang, Dong Li, and Yuxiong He. {ZeRO-Ofﬂoad}: Democratizing {Billion-Scale} Model
Training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551–564, 2021.
ISBN 978-1-939133-23-6. URL https://www.usenix.org/conference/atc21/pr
esentation/ren-jie.

Oliver Richter and Roger Wattenhofer.

Normalized Attention Without Probability Cage.

arXiv:2005.09561 [cs, stat], May 2020. URL http://arxiv.org/abs/2005.09561.

Aurko Roy, Rohan Anil, Guangda Lai, Benjamin Lee, Jeffrey Zhao, Shuyuan Zhang, Shibo
Wang, Ye Zhang, Shen Wu, Rigel Swavely, Tao, Yu, Phuong Dao, Christopher Fifty, Zhifeng
Chen, and Yonghui Wu.
N-Grammer: Augmenting Transformers with latent n-grams.
arxiv:2207.06366[cs], July 2022. doi: 10.48550/arXiv.2207.06366. URL http://arxi
v.org/abs/2207.06366.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY