15

Cerebras-GPT: Open Compute-Optimal Language Models

Noam Shazeer. GLU Variants Improve Transformer, 2020. URL https://arxiv.org/abs/2002.05202.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper,
Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using DeepSpeed and Megatron
to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model, 2022. URL https:
//arxiv.org/abs/2201.11990.

Robyn Speer. ftfy, 2019. Version 5.5.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced

Transformer with Rotary Position Embedding, 2022. URL https://arxiv.org/abs/2104.09864.

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung,
Siamak Shakeri, Dara Bahri, Tal Schuster, et al. UL2: Unifying Language Learning Paradigms, 2023.
URL http://arxiv.org/abs/2205.05131.
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster