32

Cerebras-GPT: Open Compute-Optimal Language Models

model batch size can be more ﬂexible as long as it is large enough. When tuning on a small model, Yang
and Hu choose batch size to be quite large such that it is an appropriate batch size for the largest models
to which they µTransfer those hyperparameters. We ﬁnd that proxy model tuning does not need to be
performed on a batch size appropriate for the largest model. Rather, batch size must be suﬃciently large
such that the proxy model’s gradient dynamics are likely to be consistent with the larger models. More
speciﬁcally, setting the proxy model’s batch size larger than its critical batch size (dictated by gradient noise
(McCandlish et al., 2018)) is suﬃcient to get good hyperparameter transferrability. Then, batch size should
be scaled appropriately as model size scales.

Simple Batch Size + Learning Rate Scaling with µP
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster