Recent research in large language models (LLMs) shows important advances that can improve LLM quality
and eﬃciency. Scaling law studies show predictable and signiﬁcant improvements in model performance by
increasing model and dataset size (Hestness et al., 2017; Kaplan et al., 2020). Language models can also be
improved just by training on more data (Hoﬀmann et al., 2022; Touvron et al., 2023). Recent works, such as
Maximal Update Parameterization (µP), also show techniques to improve training stability and performance
as models scale up (e.g., Bachlechner et al. (2020); Yang et al. (2021)).
Concurrently with these advances, the research community has trained and released many open-source
models. Models like GPT-J, GPT-NeoX, OPT, and Pythia have each held state-of-the-art accuracy for
open source models for their size, and these models can be tested and used simply by downloading the pre-
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster