Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the Diﬃculty of
Training Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing (EMNLP), 2020.

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Conference on

Learning Representations, 2017.

©2023 Cerebras Systems Inc. All Rights Reserved.

14

Cerebras-GPT: Open Compute-Optimal Language Models

Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An Empirical Model of Large-

Batch Training, 2018. URL https://arxiv.org/abs/1812.06162.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris
Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training.
In International Conference on Learning Representations, 2018.
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster