√
µP scales the key-query dot-product activations by 1/dhead, rather than using the 1/
originally proposed in Vaswani et al. (2017).

• The next subsection describes how we tune the three transferrable µP hyperparameters: the base
learning rate (ηbase), the base initialization standard deviation (σbase), and the embedding output
multiplier (memb).

G.2 µP Hyperparameter Search Details

We tune µP hyperparameters on a 40M parameter proxy model, and µTransfer those hyperparameters to our
models with 111M–2.7B parameters. Figure 13 shows the results of a 200 sample random hyperparameter
search on a 40M parameter proxy model (dmodel = dmodel,base = 256, nlayers = 32, dhead = 128) trained on
600M tokens with a batch size of 131k tokens. From this sweep we obtained the following tuned hyperpa-
rameters for our µP models: ηbase = 6e-3, σbase = 0.08, memb = 10. These hyperparameters also closely
match those used by Yang et al. (2021).
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster