Learning Dynamics in Continual Pre-Training for Large Language Models
Continual Pre-Training (CPT) has become a popular and effective method to
apply strong foundation models to specific downstream tasks. In this work, we
explore the learning dynamics throughout the CPT process for large language
models. We specifically focus on how general and downstream domain performance
evolves at each training step, with domain performance measured via validation
losses. We have observed that the CPT loss curve fundamentally characterizes
the transition from one curve to another hidden curve, and could be described
by decoupling the effects of distribution shift and learning rate annealing. We
derive a CPT scaling law that combines the two factors, enabling the prediction
of loss at any (continual) training steps and across learning rate schedules
(LRS) in CPT. Our formulation presents a comprehensive understanding of several
critical factors in CPT, including loss potential, peak learning rate, training
steps, replay ratio, etc. Moreover, our approach can be adapted to customize
training hyper-parameters to different CPT goals such as balancing general and
domain-specific performance. Extensive experiments demonstrate that our scaling
law holds across various CPT datasets and training hyper-parameters.
6822b079c10ac9c466c546e3
Xingjin Wang
6822b079c10ac9c466c546e4
Howe Tissue
admin_assigned
2025-05-13T08:37:40.091Z
6718fc605e14ff6b94a7109f
https://cdn-avatars.huggingface.co/v1/production/uploads/6718fc605e14ff6b94a7109f/uz2O2F_qbY3wefpY548qS.jpeg
Howe Tissue
user
Howe77
6822b079c10ac9c466c546e5
Lu Wang
claimed_verified
2025-05-18T08:47:35.031Z
67ffb89774193e65a77cd914
/avatars/4302f06bcb1fa4c372b4d675459c4068.svg
Wang Lu
user
lwzzfzl
6822b079c10ac9c466c546e6
Linjing Li
admin_assigned
2025-05-13T08:37:54.756Z
668268ace365c0f6660b67d3
/avatars/3b179b65a09dc8896b993d2013baeafb.svg
ll
user
linjinglian
6822b079c10ac9c466c546e7
Daniel Dajun Zeng
Learning Dynamics in Continual Pre-Training for Large Language Models
Continual Pre-Training (CPT) has become a popular and effective method to
apply strong foundation models to specific downstream tasks. In this work, we
explore the learning dynamics throughout the CPT process for large language
models. We specifically focus on how general and downstream domain performance
evolves at each training step, with domain performance measured via validation
losses. We have observed that the CPT loss curve fundamentally characterizes
the transition from one curve to another hidden curve, and could be described
by decoupling the effects of distribution shift and learning rate annealing. We
derive a CPT scaling law that combines the two factors, enabling the prediction
of loss at any (continual) training steps and across learning rate schedules
(LRS) in CPT. Our formulation presents a comprehensive understanding of several
critical factors in CPT, including loss potential, peak learning rate, training
steps, replay ratio, etc. Moreover, our approach can be adapted to customize
training hyper-parameters to different CPT goals such as balancing general and
domain-specific performance. Extensive experiments demonstrate that our scaling
law holds across various CPT datasets and training hyper-parameters.
The study provides a scaling law for Continual Pre-Training (CPT) of large language models, characterizing the transition in performance curves and accounting for distribution shift and learning rate changes.
Continual Pre-Training CPT large language models distribution shift learning rate annealing CPT scaling law validation losses loss potential peak learning rate training steps replay ratio