title: Language models scale reliably with over-training and on downstream
    tasks
summary: Scaling laws are useful guides for developing language models, but there are
  still gaps between current scaling studies and how language models are
  ultimately trained and evaluated. For instance, scaling is usually studied in
  the compute-optimal training regime (i.e., "Chinchilla optimal" regime);
  however, in practice, models are often over-trained to reduce inference costs.
  Moreover, scaling laws mostly predict loss on next-token prediction, but
  ultimately models are compared based on downstream task performance. In this
  paper, we address both shortcomings. To do so, we create a testbed of 104
  models with 0.011B to 6.9B parameters trained with various numbers of tokens on
  three data distributions. First, we investigate scaling in the over-trained
  regime. We fit scaling laws that extrapolate in both the number of model
  parameters and the ratio of training tokens to parameters. This enables us to
  predict the validation loss of a 1.4B parameter, 900B token run (i.e.,
  32times over-trained) and a 6.9B parameter, 138B token
  runx2014each from experiments that take 300times less compute.
  Second, we relate the perplexity of a language model to its downstream task
  performance via a power law. We use this law to predict top-1 error averaged
  over downstream tasks for the two aforementioned models using experiments that
  take 20times less compute. Our experiments are available at
  https://github.com/mlfoundations/scaling.
paper_authors._id: 65f27788f5cf26fe06294cd4
paper_authors.name: Samir Yitzhak Gadre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:15:04.958Z
paper_authors._id: 62be1cd77c0388414f012f23
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656626356583-noauth.jpeg
paper_authors.fullname: Samir Gadre
paper_authors.type: user
paper_authors.user: samirg
paper_authors._id: 65f27788f5cf26fe06294cd5
paper_authors.name: Georgios Smyrnis
paper_authors._id: 65f27788f5cf26fe06294cd6
paper_authors.name: Vaishaal Shankar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:16:18.510Z
paper_authors._id: 63118add64939fabc0108b28
paper_authors.avatarUrl: /avatars/0dfbd5d7f1111ec8798e3a9bcb5f7e77.svg
paper_authors.fullname: vaishaal shankar
paper_authors.type: user
paper_authors.user: vaishaal
paper_authors._id: 65f27788f5cf26fe06294cd7
paper_authors.name: Suchin Gururangan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:16:25.673Z
paper_authors._id: 5e9272254957053f60648a01
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1586657211685-5e9272254957053f60648a01.png
paper_authors.fullname: Suchin Gururangan
paper_authors.type: user
paper_authors.user: suching
paper_authors._id: 65f27788f5cf26fe06294cd8
paper_authors.name: Mitchell Wortsman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:17:35.484Z
paper_authors._id: 638e42f96251c8bd7ab84089
paper_authors.avatarUrl: /avatars/cb7a7f902846bd9710e2c48a50ce3527.svg
paper_authors.fullname: Mitchell Wortsman
paper_authors.type: user
paper_authors.user: mitchellw
paper_authors._id: 65f27788f5cf26fe06294cd9
paper_authors.name: Rulin Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:17:42.068Z
paper_authors._id: 6334a0bd31a2be3938c59537
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6334a0bd31a2be3938c59537/kSetFUWAmJbPQ1KSlNKBr.jpeg
paper_authors.fullname: Rulin Shao
paper_authors.type: user
paper_authors.user: rulins
paper_authors._id: 65f27788f5cf26fe06294cda
paper_authors.name: Jean Mercat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:17:49.217Z
paper_authors._id: 62be399f01dc22b4d22fc990
paper_authors.avatarUrl: /avatars/3e1f3ed30773cbb410b8a9d3c854cac7.svg
paper_authors.fullname: Jean Mercat
paper_authors.type: user
paper_authors.user: jmercat
paper_authors._id: 65f27788f5cf26fe06294cdb
paper_authors.name: Alex Fang
paper_authors._id: 65f27788f5cf26fe06294cdc
paper_authors.name: Jeffrey Li
paper_authors._id: 65f27788f5cf26fe06294cdd
paper_authors.name: Sedrick Keh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:18:21.055Z
paper_authors._id: 6365c1158b6bd0d9f6605d3d
paper_authors.avatarUrl: /avatars/1e3e6573e0cb2bb1b9d8043045120ee6.svg
paper_authors.fullname: Sedrick Keh
paper_authors.type: user
paper_authors.user: sedrickkeh
paper_authors._id: 65f27788f5cf26fe06294cde
paper_authors.name: Rui Xin
paper_authors._id: 65f27788f5cf26fe06294cdf
paper_authors.name: Marianna Nezhurina
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:22:36.319Z
paper_authors._id: 62500f361684e0335e527bc6
paper_authors.avatarUrl: /avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
paper_authors.fullname: Marianna Nezhurina
paper_authors.type: user
paper_authors.user: marianna13
paper_authors._id: 65f27788f5cf26fe06294ce0
paper_authors.name: Igor Vasiljevic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:22:57.245Z
paper_authors._id: 633b2d317af633cbcd03a53e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633b2d317af633cbcd03a53e/awtbCqUE4qBusxuu17iFg.png
paper_authors.fullname: Igor Vasiljevic
paper_authors.type: user
paper_authors.user: ivas-tri
paper_authors._id: 65f27788f5cf26fe06294ce1
paper_authors.name: Jenia Jitsev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:23:08.231Z
paper_authors._id: 6355b485b8b79340d4630dd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6355b485b8b79340d4630dd5/HIZO4ybweRy48VdCtk2MB.jpeg
paper_authors.fullname: Jenia Jitsev
paper_authors.type: user
paper_authors.user: JJitsev
paper_authors._id: 65f27788f5cf26fe06294ce2
paper_authors.name: Alexandros G. Dimakis
paper_authors._id: 65f27788f5cf26fe06294ce3
paper_authors.name: Gabriel Ilharco
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:23:21.671Z
paper_authors._id: 635c1b2986f52b24c1e9fde3
paper_authors.avatarUrl: /avatars/d53dee91892cccfd0c4d7353ffb67cbf.svg
paper_authors.fullname: Gabriel Ilharco
paper_authors.type: user
paper_authors.user: gabrielilharco
paper_authors._id: 65f27788f5cf26fe06294ce4
paper_authors.name: Shuran Song
paper_authors._id: 65f27788f5cf26fe06294ce5
paper_authors.name: Thomas Kollar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:23:53.434Z
paper_authors._id: 64358e662d0ed796668cb438
paper_authors.avatarUrl: /avatars/6f86547d8691e061e289dd906deedfdb.svg
paper_authors.fullname: Thomas Kollar
paper_authors.type: user
paper_authors.user: thomas-kollar
paper_authors._id: 65f27788f5cf26fe06294ce6
paper_authors.name: Yair Carmon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:24:48.066Z
paper_authors._id: 63fe1bd38b3c5087ff85e472
paper_authors.avatarUrl: /avatars/c481dab1044436db48c591c20441cd66.svg
paper_authors.fullname: Yair Carmon
paper_authors.type: user
paper_authors.user: ycarmon
paper_authors._id: 65f27788f5cf26fe06294ce7
paper_authors.name: Achal Dave
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:25:03.657Z
paper_authors._id: 6488e29b78570cedad71eb2d
paper_authors.avatarUrl: /avatars/0d8bb52c1e2be8d6aa7c732cf92fc5a9.svg
paper_authors.fullname: Achal Dave
paper_authors.type: user
paper_authors.user: achal-tri
paper_authors._id: 65f27788f5cf26fe06294ce8
paper_authors.name: Reinhard Heckel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:25:11.497Z
paper_authors._id: 64385800c04b3b996ea3eb3b
paper_authors.avatarUrl: /avatars/0bb710434cdb9822f861917a42c93414.svg
paper_authors.fullname: Reinhard Heckel
paper_authors.type: user
paper_authors.user: reinhardh
paper_authors._id: 65f27788f5cf26fe06294ce9
paper_authors.name: Niklas Muennighoff
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:25:20.035Z
paper_authors._id: 5f1eb362eec0ad2a071ad6e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5f1eb362eec0ad2a071ad6e2/IXMYkYKuTwn6kBdWnQeeY.png
paper_authors.fullname: Niklas Muennighoff
paper_authors.type: user
paper_authors.user: Muennighoff
paper_authors._id: 65f27788f5cf26fe06294cea
paper_authors.name: Ludwig Schmidt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:25:44.496Z
paper_authors._id: 6252375c818a5dc29ab9d945
paper_authors.avatarUrl: /avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
paper_authors.fullname: Ludwig Schmidt
paper_authors.type: user
paper_authors.user: ludwigschmidt
paper_title: Language models scale reliably with over-training and on downstream
    tasks
paper_summary: Scaling laws are useful guides for developing language models, but there are
  still gaps between current scaling studies and how language models are
  ultimately trained and evaluated. For instance, scaling is usually studied in
  the compute-optimal training regime (i.e., "Chinchilla optimal" regime);
  however, in practice, models are often over-trained to reduce inference costs.
  Moreover, scaling laws mostly predict loss on next-token prediction, but
  ultimately models are compared based on downstream task performance. In this
  paper, we address both shortcomings. To do so, we create a testbed of 104
  models with 0.011B to 6.9B parameters trained with various numbers of tokens on
  three data distributions. First, we investigate scaling in the over-trained
  regime. We fit scaling laws that extrapolate in both the number of model
  parameters and the ratio of training tokens to parameters. This enables us to
  predict the validation loss of a 1.4B parameter, 900B token run (i.e.,
  32times over-trained) and a 6.9B parameter, 138B token
  runx2014each from experiments that take 300times less compute.
  Second, we relate the perplexity of a language model to its downstream task
  performance via a power law. We use this law to predict top-1 error averaged
  over downstream tasks for the two aforementioned models using experiments that
  take 20times less compute. Our experiments are available at
  https://github.com/mlfoundations/scaling.
paper_ai_keywords: compute-optimal training regimes Chinchilla optimal regime over-trained inference costs scaling laws testbed model parameters training tokens data distributions over-trained regime validation loss perplexity downstream task performance top-1 error