Early deep learning scaling law studies show that when scaling dataset and model size, loss improves pre-
dictably (Hestness et al., 2017; Kaplan et al., 2020). These studies indicate generally that scaling could give
substantial modeling improvements. From this observation, many organizations scaled to train the largest
possible models on their available infrastructure: GPT-3 175B (Brown et al., 2020), Jurassic-1 178B (Lieber
et al., 2021), Gopher 280B (Rae et al., 2022), HyperCLOVA 82B (Kim et al., 2021), Ernie 3.0 Titan 260B
(Wang et al., 2021), Yuan 1.0 (Wu et al., 2021), PanGu-α (Zeng et al., 2021), Megatron-Turing NLG 530B
(Smith et al., 2022), PaLM 540B (Chowdhery et al., 2022), and LaMDA 137B (Thoppilan et al., 2022). These
models show signiﬁcant performance improvement on many downstream tasks compared to prior language
models. However, these studies typically only scale model size without scaling the dataset size as suggested
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster