title: Simple and Scalable Strategies to Continually Pre-train Large Language
    Models
summary: Large language models (LLMs) are routinely pre-trained on billions of tokens,
  only to start the process over again once new data becomes available. A much
  more efficient solution is to continually pre-train these models, saving
  significant compute compared to re-training. However, the distribution shift
  induced by new data typically results in degraded performance on previous data
  or poor adaptation to the new data. In this work, we show that a simple and
  scalable combination of learning rate (LR) re-warming, LR re-decaying, and
  replay of previous data is sufficient to match the performance of fully
  re-training from scratch on all available data, as measured by final loss and
  language model (LM) evaluation benchmarks. Specifically, we show this for a
  weak but realistic distribution shift between two commonly used LLM
  pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution
  shift (EnglishrightarrowGerman) at the 405M parameter model scale with
  large dataset sizes (hundreds of billions of tokens). Selecting the weak but
  realistic shift for larger-scale experiments, we also find that our continual
  learning strategies match the re-training baseline for a 10B parameter LLM. Our
  results demonstrate that LLMs can be successfully updated via simple and
  scalable continual learning strategies, matching the re-training baseline using
  only a fraction of the compute. Finally, inspired by previous work, we propose
  alternatives to the cosine learning rate schedule that help circumvent
  forgetting induced by LR re-warming and that are not bound to a fixed token
  budget.
paper_authors._id: 65f271e1b06ca1ff48a12ff3
paper_authors.name: Adam Ibrahim
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-03-14T04:44:57.674Z
paper_authors._id: 648743f05aed4a7b0bf2e3a6
paper_authors.avatarUrl: /avatars/e6f3b35085b1b8cb2c231516d1fea70a.svg
paper_authors.fullname: Adam Ibrahim
paper_authors.type: user
paper_authors.user: adami1
paper_authors._id: 65f271e1b06ca1ff48a12ff4
paper_authors.name: Benjamin Thérien
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:01:31.759Z
paper_authors._id: 62d0e2a3fb896639b2a04ca1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d0e2a3fb896639b2a04ca1/UBZzdldkvCAEHp-t6L2JJ.jpeg
paper_authors.fullname: Benjamin Therien
paper_authors.type: user
paper_authors.user: btherien
paper_authors._id: 65f271e1b06ca1ff48a12ff5
paper_authors.name: Kshitij Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-14T07:59:44.947Z
paper_authors._id: 625daa0c943e346492b63d21
paper_authors.avatarUrl: /avatars/860f8e9bff85325096b6f8e15102def6.svg
paper_authors.fullname: Kshitij Gupta
paper_authors.type: user
paper_authors.user: kshitijkg
paper_authors._id: 65f271e1b06ca1ff48a12ff6
paper_authors.name: Mats L. Richter
paper_authors._id: 65f271e1b06ca1ff48a12ff7
paper_authors.name: Quentin Anthony
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-20T01:09:03.116Z
paper_authors._id: 63b9a261efe99543b34e9579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b9a261efe99543b34e9579/aZMhhnurORuPU_WryDUiu.jpeg
paper_authors.fullname: Quentin Anthony
paper_authors.type: user
paper_authors.user: qanthony
paper_authors._id: 65f271e1b06ca1ff48a12ff8
paper_authors.name: Timothée Lesort
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-14T03:41:24.077Z
paper_authors._id: 631610b876c93113fb129947
paper_authors.avatarUrl: /avatars/fe80d40ac13330c2cb64958d2aee253d.svg
paper_authors.fullname: Timothée Lesort
paper_authors.type: user
paper_authors.user: TLESORT
paper_authors._id: 65f271e1b06ca1ff48a12ff9
paper_authors.name: Eugene Belilovsky
paper_authors._id: 65f271e1b06ca1ff48a12ffa
paper_authors.name: Irina Rish
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-14T08:02:13.169Z
paper_authors._id: 6227f764236b7b2eefa7f173
paper_authors.avatarUrl: /avatars/0204e3da8e9ef12d88c98c6e955f8340.svg
paper_authors.fullname: Irina Rish
paper_authors.type: user
paper_authors.user: irinarish
paper_title: Simple and Scalable Strategies to Continually Pre-train Large Language
    Models
paper_summary: Large language models (LLMs) are routinely pre-trained on billions of tokens,
  only to start the process over again once new data becomes available. A much
  more efficient solution is to continually pre-train these models, saving
  significant compute compared to re-training. However, the distribution shift
  induced by new data typically results in degraded performance on previous data
  or poor adaptation to the new data. In this work, we show that a simple and
  scalable combination of learning rate (LR) re-warming, LR re-decaying, and
  replay of previous data is sufficient to match the performance of fully
  re-training from scratch on all available data, as measured by final loss and
  language model (LM) evaluation benchmarks. Specifically, we show this for a
  weak but realistic distribution shift between two commonly used LLM
  pre-training datasets (EnglishrightarrowEnglish) and a stronger distribution
  shift (EnglishrightarrowGerman) at the 405M parameter model scale with
  large dataset sizes (hundreds of billions of tokens). Selecting the weak but
  realistic shift for larger-scale experiments, we also find that our continual
  learning strategies match the re-training baseline for a 10B parameter LLM. Our
  results demonstrate that LLMs can be successfully updated via simple and
  scalable continual learning strategies, matching the re-training baseline using
  only a fraction of the compute. Finally, inspired by previous work, we propose
  alternatives to the cosine learning rate schedule that help circumvent
  forgetting induced by LR re-warming and that are not bound to a fixed token
  budget.
paper_ai_summary: Simple continual learning strategies, including learning rate re-warming, re-decaying, and data replay, can match the performance of full re-training for large language models with much lower computational cost.
paper_ai_keywords: learning rate re-warming learning rate re-decaying data replay continual learning distribution shift language model evaluation benchmarks