title: Algorithmic progress in language models
summary: We investigate the rate at which algorithms for pre-training language models
  have improved since the advent of deep learning. Using a dataset of over 200
  language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we
  find that the compute required to reach a set performance threshold has halved
  approximately every 8 months, with a 95% confidence interval of around 5 to 14
  months, substantially faster than hardware gains per Moore's Law. We estimate
  augmented scaling laws, which enable us to quantify algorithmic progress and
  determine the relative contributions of scaling models versus innovations in
  training algorithms. Despite the rapid pace of algorithmic progress and the
  development of new architectures such as the transformer, our analysis reveals
  that the increase in compute made an even larger contribution to overall
  performance improvements over this time period. Though limited by noisy
  benchmark data, our analysis quantifies the rapid progress in language
  modeling, shedding light on the relative contributions from compute and
  algorithms.
paper_authors._id: 65efcbd5f62dffb225ce0e35
paper_authors.name: Anson Ho
paper_authors._id: 65efcbd5f62dffb225ce0e36
paper_authors.name: Tamay Besiroglu
paper_authors._id: 65efcbd5f62dffb225ce0e37
paper_authors.name: Ege Erdil
paper_authors._id: 65efcbd5f62dffb225ce0e38
paper_authors.name: David Owen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:37:31.164Z
paper_authors._id: 646f5bf8075e11ca78dbb4e5
paper_authors.avatarUrl: /avatars/7dea81e7e8f4be92e99ebcfa39150dc2.svg
paper_authors.fullname: David Owen
paper_authors.type: user
paper_authors.user: davidowen-epoch
paper_authors._id: 65efcbd5f62dffb225ce0e39
paper_authors.name: Robi Rahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:37:16.997Z
paper_authors._id: 6552600d1a60bb531b61025e
paper_authors.avatarUrl: /avatars/36c7c126244e5f25ae65f7e8562a89c2.svg
paper_authors.fullname: Robi Rahman
paper_authors.type: user
paper_authors.user: robi-epoch
paper_authors._id: 65efcbd5f62dffb225ce0e3a
paper_authors.name: Zifan Carl Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:37:41.899Z
paper_authors._id: 6333d2ff518c0f78a6e6f8e2
paper_authors.avatarUrl: /avatars/b67cda187d941341794c9da3c7dc3b39.svg
paper_authors.fullname: Zifan Guo
paper_authors.type: user
paper_authors.user: luminolblue
paper_authors._id: 65efcbd5f62dffb225ce0e3b
paper_authors.name: David Atkinson
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T14:48:57.192Z
paper_authors._id: 64769e3c3d8d1cb791909f09
paper_authors.avatarUrl: /avatars/c09f6f1df52a710db6a65e37a4545458.svg
paper_authors.fullname: David Atkinson
paper_authors.type: user
paper_authors.user: datkinson
paper_authors._id: 65efcbd5f62dffb225ce0e3c
paper_authors.name: Neil Thompson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:38:13.562Z
paper_authors._id: 636ec54587545ca5a13a4997
paper_authors.avatarUrl: /avatars/d3d7ba9515d0d7c02a92e6e914dea1ec.svg
paper_authors.fullname: Neil Thompson
paper_authors.type: user
paper_authors.user: neilqvt
paper_authors._id: 65efcbd5f62dffb225ce0e3d
paper_authors.name: Jaime Sevilla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:38:19.528Z
paper_authors._id: 60d2e60d07da9c17c72708f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624434395907-60d2e60d07da9c17c72708f9.png
paper_authors.fullname: Jaime Sevilla
paper_authors.type: user
paper_authors.user: Jsevillamol
paper_title: Algorithmic progress in language models
paper_summary: We investigate the rate at which algorithms for pre-training language models
  have improved since the advent of deep learning. Using a dataset of over 200
  language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we
  find that the compute required to reach a set performance threshold has halved
  approximately every 8 months, with a 95% confidence interval of around 5 to 14
  months, substantially faster than hardware gains per Moore's Law. We estimate
  augmented scaling laws, which enable us to quantify algorithmic progress and
  determine the relative contributions of scaling models versus innovations in
  training algorithms. Despite the rapid pace of algorithmic progress and the
  development of new architectures such as the transformer, our analysis reveals
  that the increase in compute made an even larger contribution to overall
  performance improvements over this time period. Though limited by noisy
  benchmark data, our analysis quantifies the rapid progress in language
  modeling, shedding light on the relative contributions from compute and
  algorithms.
paper_ai_summary: Analysis of language model evaluations shows that compute requirements halve faster than hardware improvements, with compute making the larger contribution to performance gains.
paper_ai_keywords: pre-training language models deep learning dataset Wikitext Penn Treebank compute performance threshold Moore's Law augmented scaling laws training algorithms transformer language modeling