title: Learning Dynamics in Continual Pre-Training for Large Language Models
summary: Continual Pre-Training (CPT) has become a popular and effective method to
  apply strong foundation models to specific downstream tasks. In this work, we
  explore the learning dynamics throughout the CPT process for large language
  models. We specifically focus on how general and downstream domain performance
  evolves at each training step, with domain performance measured via validation
  losses. We have observed that the CPT loss curve fundamentally characterizes
  the transition from one curve to another hidden curve, and could be described
  by decoupling the effects of distribution shift and learning rate annealing. We
  derive a CPT scaling law that combines the two factors, enabling the prediction
  of loss at any (continual) training steps and across learning rate schedules
  (LRS) in CPT. Our formulation presents a comprehensive understanding of several
  critical factors in CPT, including loss potential, peak learning rate, training
  steps, replay ratio, etc. Moreover, our approach can be adapted to customize
  training hyper-parameters to different CPT goals such as balancing general and
  domain-specific performance. Extensive experiments demonstrate that our scaling
  law holds across various CPT datasets and training hyper-parameters.
paper_authors._id: 6822b079c10ac9c466c546e3
paper_authors.name: Xingjin Wang
paper_authors._id: 6822b079c10ac9c466c546e4
paper_authors.name: Howe Tissue
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:37:40.091Z
paper_authors._id: 6718fc605e14ff6b94a7109f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6718fc605e14ff6b94a7109f/uz2O2F_qbY3wefpY548qS.jpeg
paper_authors.fullname: Howe Tissue
paper_authors.type: user
paper_authors.user: Howe77
paper_authors._id: 6822b079c10ac9c466c546e5
paper_authors.name: Lu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-18T08:47:35.031Z
paper_authors._id: 67ffb89774193e65a77cd914
paper_authors.avatarUrl: /avatars/4302f06bcb1fa4c372b4d675459c4068.svg
paper_authors.fullname: Wang Lu
paper_authors.type: user
paper_authors.user: lwzzfzl
paper_authors._id: 6822b079c10ac9c466c546e6
paper_authors.name: Linjing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:37:54.756Z
paper_authors._id: 668268ace365c0f6660b67d3
paper_authors.avatarUrl: /avatars/3b179b65a09dc8896b993d2013baeafb.svg
paper_authors.fullname: ll
paper_authors.type: user
paper_authors.user: linjinglian
paper_authors._id: 6822b079c10ac9c466c546e7
paper_authors.name: Daniel Dajun Zeng
paper_title: Learning Dynamics in Continual Pre-Training for Large Language Models
paper_summary: Continual Pre-Training (CPT) has become a popular and effective method to
  apply strong foundation models to specific downstream tasks. In this work, we
  explore the learning dynamics throughout the CPT process for large language
  models. We specifically focus on how general and downstream domain performance
  evolves at each training step, with domain performance measured via validation
  losses. We have observed that the CPT loss curve fundamentally characterizes
  the transition from one curve to another hidden curve, and could be described
  by decoupling the effects of distribution shift and learning rate annealing. We
  derive a CPT scaling law that combines the two factors, enabling the prediction
  of loss at any (continual) training steps and across learning rate schedules
  (LRS) in CPT. Our formulation presents a comprehensive understanding of several
  critical factors in CPT, including loss potential, peak learning rate, training
  steps, replay ratio, etc. Moreover, our approach can be adapted to customize
  training hyper-parameters to different CPT goals such as balancing general and
  domain-specific performance. Extensive experiments demonstrate that our scaling
  law holds across various CPT datasets and training hyper-parameters.
paper_ai_summary: The study provides a scaling law for Continual Pre-Training (CPT) of large language models, characterizing the transition in performance curves and accounting for distribution shift and learning rate changes.
paper_ai_keywords: Continual Pre-Training CPT large language models distribution shift learning rate annealing CPT scaling law validation losses loss potential peak learning rate training steps replay ratio