title: Efficient Continual Pre-training by Mitigating the Stability Gap
summary: Continual pre-training has increasingly become the predominant approach for
  adapting Large Language Models (LLMs) to new domains. This process involves
  updating the pre-trained LLM with a corpus from a new domain, resulting in a
  shift in the training distribution. To study the behavior of LLMs during this
  shift, we measured the model's performance throughout the continual
  pre-training process. we observed a temporary performance drop at the
  beginning, followed by a recovery phase, a phenomenon known as the "stability
  gap," previously noted in vision models classifying new classes. To address
  this issue and enhance LLM performance within a fixed compute budget, we
  propose three effective strategies: (1) Continually pre-training the LLM on a
  subset with a proper size for multiple epochs, resulting in faster performance
  recovery than pre-training the LLM on a large corpus in a single epoch; (2)
  Pre-training the LLM only on high-quality sub-corpus, which rapidly boosts
  domain performance; and (3) Using a data mixture similar to the pre-training
  data to reduce distribution gap. We conduct various experiments on Llama-family
  models to validate the effectiveness of our strategies in both medical
  continual pre-training and instruction tuning. For example, our strategies
  improve the average medical task performance of the OpenLlama-3B model from
  36.2% to 40.7% with only 40% of the original training budget and enhance the
  average general task performance without causing forgetting. Furthermore, we
  apply our strategies to the Llama-3-8B model. The resulting model,
  Llama-3-Physician, achieves the best medical performance among current
  open-source models, and performs comparably to or even better than GPT-4 on
  several medical benchmarks. We release our models at
  https://huggingface.co/YiDuo1999/Llama-3-Physician-8B-Instruct.
paper_authors._id: 667a35b89f78b2d01bd1b5b7
paper_authors.name: Yiduo Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:28:43.723Z
paper_authors._id: 64e3282156b920ef00cf7d94
paper_authors.avatarUrl: /avatars/8a76c27fe6c71eb98a28b4ebfb90336d.svg
paper_authors.fullname: Guo
paper_authors.type: user
paper_authors.user: YiDuo1999
paper_authors._id: 667a35b89f78b2d01bd1b5b8
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T08:58:55.208Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 667a35b89f78b2d01bd1b5b9
paper_authors.name: Huishuai Zhang
paper_authors._id: 667a35b89f78b2d01bd1b5ba
paper_authors.name: Dongyan Zhao
paper_authors._id: 667a35b89f78b2d01bd1b5bb
paper_authors.name: Yikang Shen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-25T18:40:38.691Z
paper_authors._id: 6463995939359568c63d740c
paper_authors.avatarUrl: /avatars/11caacefc68c1259dbc8764cc4340481.svg
paper_authors.fullname: Yikang Shen
paper_authors.type: user
paper_authors.user: YikangS
paper_title: Efficient Continual Pre-training by Mitigating the Stability Gap
paper_summary: Continual pre-training has increasingly become the predominant approach for
  adapting Large Language Models (LLMs) to new domains. This process involves
  updating the pre-trained LLM with a corpus from a new domain, resulting in a
  shift in the training distribution. To study the behavior of LLMs during this
  shift, we measured the model's performance throughout the continual
  pre-training process. we observed a temporary performance drop at the
  beginning, followed by a recovery phase, a phenomenon known as the "stability
  gap," previously noted in vision models classifying new classes. To address
  this issue and enhance LLM performance within a fixed compute budget, we
  propose three effective strategies: (1) Continually pre-training the LLM on a
  subset with a proper size for multiple epochs, resulting in faster performance
  recovery than pre-training the LLM on a large corpus in a single epoch; (2)
  Pre-training the LLM only on high-quality sub-corpus, which rapidly boosts
  domain performance; and (3) Using a data mixture similar to the pre-training
  data to reduce distribution gap. We conduct various experiments on Llama-family
  models to validate the effectiveness of our strategies in both medical
  continual pre-training and instruction tuning. For example, our strategies
  improve the average medical task performance of the OpenLlama-3B model from
  36.2% to 40.7% with only 40% of the original training budget and enhance the
  average general task performance without causing forgetting. Furthermore, we
  apply our strategies to the Llama-3-8B model. The resulting model,
  Llama-3-Physician, achieves the best medical performance among current
  open-source models, and performs comparably to or even better than GPT-4 on
  several medical benchmarks. We release our models at
  https://huggingface.co/YiDuo1999/Llama-3-Physician-8B-Instruct.
paper_ai_summary: Strategies for continual pre-training of large language models reduce performance drops and computational costs during adaptation to new domains.
paper_ai_keywords: Large Language Models continual pre-training stability gap vision models medical task performance OpenLlama-3B Llama-3-8B Llama-3-Physician GPT-4 medical benchmarks