In this section, we show pre-training and downstream evaluations of Cerebras-GPT models, scaled from
111M to 13B parameters, and we compare against recent related works. We characterize the compute-
eﬃcient Pareto frontier for pre-training models on the Pile dataset and show that models on this frontier
are also competitive on downstream tasks. We believe this is the ﬁrst study to release a compute-optimal
scaling law for pre-training on the Pile dataset that is openly reproducible by the community.
We show that Cerebras-GPT models deﬁne the state-of-the-art compute-optimal Pareto frontier on both
pre-training and downstream objectives. Further, our largest model with 13B parameters shows improved
accuracy on most downstream tasks compared to other comparably-sized publicly-available models1. We also
train Cerebras-GPT models conﬁgured using µP. We show that µP enables direct hyperparameter transfer
from smaller to larger models and improves the compute-optimal frontier loss by 0.4%.
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster