title: Unlocking Continual Learning Abilities in Language Models
summary: Language models (LMs) exhibit impressive performance and generalization
  capabilities. However, LMs struggle with the persistent challenge of
  catastrophic forgetting, which undermines their long-term sustainability in
  continual learning (CL). Existing approaches usually address the issue by
  incorporating old task data or task-wise inductive bias into LMs. However, old
  data and accurate task information are often unavailable or costly to collect,
  hindering the availability of current CL approaches for LMs. To address this
  limitation, we introduce MIGU (MagnItude-based
  Gradient Updating for continual learning), a
  rehearsal-free and task-label-free method that only updates the model
  parameters with large magnitudes of output in LMs' linear layers. MIGU is based
  on our observation that the L1-normalized magnitude distribution of the output
  in LMs' linear layers is different when the LM models deal with different task
  data. By imposing this simple constraint on the gradient update process, we can
  leverage the inherent behaviors of LMs, thereby unlocking their innate CL
  abilities. Our experiments demonstrate that MIGU is universally applicable to
  all three LM architectures (T5, RoBERTa, and Llama2), delivering
  state-of-the-art or on-par performance across continual finetuning and
  continual pre-training settings on four CL benchmarks. For example, MIGU brings
  a 15.2% average accuracy improvement over conventional parameter-efficient
  finetuning baselines in a 15-task CL benchmark. MIGU can also seamlessly
  integrate with all three existing CL types to further enhance performance. Code
  is available at https://github.com/wenyudu/MIGU{this https URL}.
paper_authors._id: 667b8e9481adc76dc729773d
paper_authors.name: Wenyu Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:03:22.268Z
paper_authors._id: 624c3d2ca19f20b197761ba9
paper_authors.avatarUrl: /avatars/7a64b81c29f4f6700fa18effc5616865.svg
paper_authors.fullname: Wenyu Du
paper_authors.type: user
paper_authors.user: wydu
paper_authors._id: 667b8e9481adc76dc729773e
paper_authors.name: Shuang Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-15T10:12:44.896Z
paper_authors._id: 63478bc1e5c0717e673a086d
paper_authors.avatarUrl: /avatars/66513149f2e92a65bec54364c2d360c0.svg
paper_authors.fullname: Cheng Shuang
paper_authors.type: user
paper_authors.user: chengs18
paper_authors._id: 667b8e9481adc76dc729773f
paper_authors.name: Tongxu Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T10:18:26.315Z
paper_authors._id: 6421b07e918f0fd889f0a682
paper_authors.avatarUrl: /avatars/314b55c2428426c846d9449f98db4355.svg
paper_authors.fullname: Tongxu Luo
paper_authors.type: user
paper_authors.user: Kuroxiro
paper_authors._id: 667b8e9481adc76dc7297740
paper_authors.name: Zihan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T12:29:42.663Z
paper_authors._id: 647ccbd6e07cf9bb2d485244
paper_authors.avatarUrl: /avatars/e8915abaff04f6762247e196b7cf84df.svg
paper_authors.fullname: Zihan Qiu
paper_authors.type: user
paper_authors.user: QwQZh
paper_authors._id: 667b8e9481adc76dc7297741
paper_authors.name: Zeyu Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T12:42:36.119Z
paper_authors._id: 64a5520dea3c5861303064b0
paper_authors.avatarUrl: /avatars/7e618bcfc4d8d3291cf9bc8f2c4c1b15.svg
paper_authors.fullname: Zeyu hUANG
paper_authors.type: user
paper_authors.user: zeroyhuang
paper_authors._id: 667b8e9481adc76dc7297742
paper_authors.name: Ka Chun Cheung
paper_authors._id: 667b8e9481adc76dc7297743
paper_authors.name: Reynold Cheng
paper_authors._id: 667b8e9481adc76dc7297744
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:03:19.335Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/dFR_ofjbqCrcqGa9R3MMq.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_title: Unlocking Continual Learning Abilities in Language Models
paper_summary: Language models (LMs) exhibit impressive performance and generalization
  capabilities. However, LMs struggle with the persistent challenge of
  catastrophic forgetting, which undermines their long-term sustainability in
  continual learning (CL). Existing approaches usually address the issue by
  incorporating old task data or task-wise inductive bias into LMs. However, old
  data and accurate task information are often unavailable or costly to collect,
  hindering the availability of current CL approaches for LMs. To address this
  limitation, we introduce MIGU (MagnItude-based
  Gradient Updating for continual learning), a
  rehearsal-free and task-label-free method that only updates the model
  parameters with large magnitudes of output in LMs' linear layers. MIGU is based
  on our observation that the L1-normalized magnitude distribution of the output
  in LMs' linear layers is different when the LM models deal with different task
  data. By imposing this simple constraint on the gradient update process, we can
  leverage the inherent behaviors of LMs, thereby unlocking their innate CL
  abilities. Our experiments demonstrate that MIGU is universally applicable to
  all three LM architectures (T5, RoBERTa, and Llama2), delivering
  state-of-the-art or on-par performance across continual finetuning and
  continual pre-training settings on four CL benchmarks. For example, MIGU brings
  a 15.2% average accuracy improvement over conventional parameter-efficient
  finetuning baselines in a 15-task CL benchmark. MIGU can also seamlessly
  integrate with all three existing CL types to further enhance performance. Code
  is available at https://github.com/wenyudu/MIGU{this https URL}.
paper_ai_keywords: catastrophic forgetting continual learning (CL) rehearsal-free task-label-free gradient updating L1-normalized magnitude distribution continual finetuning continual pre-training T5 RoBERTa Llama2 parameter-efficient finetuning