• We train Cerebras-GPT compute-optimal models scaled from 111M to 13B parameters on the Pile

dataset following Chinchilla scaling rules to collect compute-eﬃcient scaling laws.

• We show that these models provide state-of-the-art pre-training eﬃciency on both pre-training and

downstream objectives compared to other open models–the ﬁrst such open eﬀort.

• We provide detailed instructions to reproduce our results, including the use of µP to improve training

stability and transfer hyperparameters as models scale up.

• We document our experience training these models on the Andromeda AI Cluster, comprising 16

Cerebras CS-2 systems, and we describe the simplicity of scaling models and performance.
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster