As we scaled the Cerebras-GPT models with standard parameterization (SP) along our scaling law, we ex-
perienced challenges predicting appropriate hyperparameters, and these models show substantial variance
around their common scaling law. To address these challenges, we also test µP and µTransfer tuned hyper-
parameters to 111M–2.7B parameter Cerebras-GPT models. Across model sizes, our µP models exhibit an
average of 0.43% improved Pile test loss and 1.7% higher average downstream task accuracy compared to
our SP models. Here, we also show that µP performance scales more predictably, enabling more accurate
performance extrapolation.
As we scaled up models with SP, we found model training could become unstable when conﬁgured with
hyperparameters used in other prior works. At diﬀerent model size scales, the numerical characteristics of
diﬀerent layers can training instability. These instabilities can lead the practitioner to adjust prior hyper-
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster