We study recent research advances that improve large language models through eﬃcient pre-
training and scaling, and open datasets and tools. We combine these advances to introduce
Cerebras-GPT, a family of open compute-optimal language models scaled from 111M to
13B parameters. We train Cerebras-GPT models on the Eleuther Pile dataset following
DeepMind Chinchilla scaling rules for eﬃcient pre-training (highest accuracy for a given
compute budget). We characterize the predictable power-law scaling and compare Cerebras-
GPT with other publicly-available models to show all Cerebras-GPT models have state-of-
the-art training eﬃciency on both pre-training and downstream objectives. We describe
our learnings including how Maximal Update Parameterization (µP) can further improve
large model scaling, improving accuracy and hyperparameter predictability at scale. We
release our pre-trained models and code, making this paper the ﬁrst open and reproducible
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster