2.4 Standard (SP) and Maximal Update Parameterization (µP)

Standard Parameterization (SP): We conﬁgure our main Cerebras-GPT models with the common stan-
dard parameterization (SP) approach. In SP, model weights are initialized from normal distributions with
constant standard deviation or standard deviation based on the shape of each layer (Glorot & Bengio, 2010).
We initialize embedding and hidden layer weights with a truncated normal distribution with standard devi-

ation σ = 0.02. An exception is that we use a standard deviation of σ = 0.02/p2 · nlayers for the last layer
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster