We train models using the following training conﬁgurations. We use the AdamW optimizer (Loshchilov &
Hutter, 2017) with (beta1, beta2) = (0.9, 0.95). We set epsilon to 1e-8 for small models and to 1e-9 for
6.7B and 13B parameter models. We use weight decay of 0.1 for all models. We do not use dropout for
pre-training. For all runs, we use gradient norm clipping of 1.0.
We use learning rates and batch sizes consistent with prior works, as listed in Table 1. We ﬁnd that linear
learning rate decay tends to perform better than cosine decay, so we use it in most of our pre-training runs.
With either decay type, we warm up learning rate linearly over 375M tokens and then decay to 10% of the
maximum learning rate. The table also includes batch sizing. For the 13B parameter model, we train with a
batch size of 720 sequences of length 2048 tokens for the ﬁrst 84B tokens. At that point, we observed the gap
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster