title: Control LLM: Controlled Evolution for Intelligence Retention in LLM
summary: Large Language Models (LLMs) demand significant computational resources,
  making it essential to enhance their capabilities without retraining from
  scratch. A key challenge in this domain is catastrophic forgetting
  (CF), which hampers performance during Continuous Pre-training (CPT) and
  Continuous Supervised Fine-Tuning (CSFT). We propose Control LLM, a
  novel approach that leverages parallel pre-trained and expanded transformer
  blocks, aligning their hidden-states through interpolation strategies This
  method effectively preserves performance on existing tasks while seamlessly
  integrating new knowledge.
    Extensive experiments demonstrate the effectiveness of Control LLM in both
  CPT and CSFT. On Llama3.1-8B-Instruct, it achieves significant improvements in
  mathematical reasoning (+14.4% on Math-Hard) and coding performance (+10%
  on MBPP-PLUS). On Llama3.1-8B, it enhances multilingual capabilities (+10.6%
  on C-Eval, +6.8% on CMMLU, and +30.2% on CMMLU-0shot-CoT). It surpasses
  existing methods and achieves SOTA among open-source models tuned from the same
  base model, using substantially less data and compute. Crucially, these gains
  are realized while preserving strong original capabilities, with minimal
  degradation (<4.3% on MMLU) compared to >35% in open-source Math
  and Coding models. This approach has been successfully deployed in LinkedIn's
  GenAI-powered job seeker and Ads unit products.
    To support further research, we release the training and evaluation code
  (https://github.com/linkedin/ControlLLM) along with models trained on
  public datasets ( https://huggingface.co/ControlLLM) to the community.
paper_authors._id: 6792cf69627180db59a51b7c
paper_authors.name: Haichao Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-24T09:08:11.344Z
paper_authors._id: 6199b1d090692f0d92388473
paper_authors.avatarUrl: /avatars/72693e4d37ded1542cd2564879fd9a61.svg
paper_authors.fullname: Haichao Wei
paper_authors.type: user
paper_authors.user: hawei
paper_authors._id: 6792cf69627180db59a51b7d
paper_authors.name: Yunxiang Ren
paper_authors._id: 6792cf69627180db59a51b7e
paper_authors.name: Zhoutong Fu
paper_authors._id: 6792cf69627180db59a51b7f
paper_authors.name: Aman Lunia
paper_authors._id: 6792cf69627180db59a51b80
paper_authors.name: Yi-Lin Chen
paper_authors._id: 6792cf69627180db59a51b81
paper_authors.name: Alice Leung
paper_authors._id: 6792cf69627180db59a51b82
paper_authors.name: Ya Xu
paper_title: Control LLM: Controlled Evolution for Intelligence Retention in LLM
paper_summary: Large Language Models (LLMs) demand significant computational resources,
  making it essential to enhance their capabilities without retraining from
  scratch. A key challenge in this domain is catastrophic forgetting
  (CF), which hampers performance during Continuous Pre-training (CPT) and
  Continuous Supervised Fine-Tuning (CSFT). We propose Control LLM, a
  novel approach that leverages parallel pre-trained and expanded transformer
  blocks, aligning their hidden-states through interpolation strategies This
  method effectively preserves performance on existing tasks while seamlessly
  integrating new knowledge.
    Extensive experiments demonstrate the effectiveness of Control LLM in both
  CPT and CSFT. On Llama3.1-8B-Instruct, it achieves significant improvements in
  mathematical reasoning (+14.4% on Math-Hard) and coding performance (+10%
  on MBPP-PLUS). On Llama3.1-8B, it enhances multilingual capabilities (+10.6%
  on C-Eval, +6.8% on CMMLU, and +30.2% on CMMLU-0shot-CoT). It surpasses
  existing methods and achieves SOTA among open-source models tuned from the same
  base model, using substantially less data and compute. Crucially, these gains
  are realized while preserving strong original capabilities, with minimal
  degradation (<4.3% on MMLU) compared to >35% in open-source Math
  and Coding models. This approach has been successfully deployed in LinkedIn's
  GenAI-powered job seeker and Ads unit products.
    To support further research, we release the training and evaluation code
  (https://github.com/linkedin/ControlLLM) along with models trained on
  public datasets ( https://huggingface.co/ControlLLM) to the community.
paper_ai_keywords: catastrophic forgetting Continuous Pre-training Continuous Supervised Fine-Tuning Control LLM parallel pre-trained and expanded transformer blocks interpolation strategies mathematical robust MBPP-PLUS multilingual capabilities C-Eval CMMLU CMMLU-0shot-CoT MMLU