Language models scale reliably with over-training and on downstream tasks
<|reference_start|>Language models scale reliably with over-training and on downstream tasks: Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., "Chinchilla optimal" regime). In contrast, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance. To address both shortcomings, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32$\times$ over-trained) and a 6.9B parameter, 138B token run (i.e., a compute-optimal run)$\unicode{x2014}$each from experiments that take 300$\times$ less compute. Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20$\times$ less compute. Our experiments are available at https://github.com/mlfoundations/scaling.<|reference_end|>
@article{gadre2024language,
    title={Language models scale reliably with over-training and on downstream
  tasks},
    author={Samir Yitzhak Gadre and Georgios Smyrnis and Vaishaal Shankar and
  Suchin Gururangan and Mitchell Wortsman and Rulin Shao and Jean Mercat and
  Alex Fang and Jeffrey Li and Sedrick Keh and Rui Xin and Marianna Nezhurina
  and Igor Vasiljevic and Jenia Jitsev and Luca Soldaini and Alexandros G.
  Dimakis and Gabriel Ilharco and Pang Wei Koh and Shuran Song and Thomas
  Kollar and Yair Carmon and Achal Dave and Reinhard Heckel and Niklas
  Muennighoff and Ludwig Schmidt},
    journal={arXiv preprint arXiv:2403.08540},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.08540},
    primaryClass={cs.CL cs.LG}
}