title: Towards Robust and Efficient Continual Language Learning
summary: As the application space of language models continues to evolve, a natural
  question to ask is how we can quickly adapt models to new tasks. We approach
  this classic question from a continual learning perspective, in which we aim to
  continue fine-tuning models trained on past tasks on new tasks, with the goal
  of "transferring" relevant knowledge. However, this strategy also runs the risk
  of doing more harm than good, i.e., negative transfer. In this paper, we
  construct a new benchmark of task sequences that target different possible
  transfer scenarios one might face, such as a sequence of tasks with high
  potential of positive transfer, high potential for negative transfer, no
  expected effect, or a mixture of each. An ideal learner should be able to
  maximally exploit information from all tasks that have any potential for
  positive transfer, while also avoiding the negative effects of any distracting
  tasks that may confuse it. We then propose a simple, yet effective, learner
  that satisfies many of our desiderata simply by leveraging a selective strategy
  for initializing new models from past task checkpoints. Still, limitations
  remain, and we hope this benchmark can help the community to further build and
  analyze such learners.
paper_authors._id: 64af5c882cda6a37a48d04bd
paper_authors.name: Adam Fisch
paper_authors._id: 64af5c882cda6a37a48d04be
paper_authors.name: Amal Rannen-Triki
paper_authors._id: 64af5c882cda6a37a48d04bf
paper_authors.name: Razvan Pascanu
paper_authors._id: 64af5c882cda6a37a48d04c0
paper_authors.name: Jörg Bornschein
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T08:58:56.944Z
paper_authors._id: 6679bdb03043e7ed261a7699
paper_authors.avatarUrl: /avatars/482ce2c4b2daf194161a0210e66d96de.svg
paper_authors.fullname: Jörg Bornschein
paper_authors.type: user
paper_authors.user: jbornschein
paper_authors._id: 64af5c882cda6a37a48d04c1
paper_authors.name: Angeliki Lazaridou
paper_authors._id: 64af5c882cda6a37a48d04c2
paper_authors.name: Elena Gribovskaya
paper_authors._id: 64af5c882cda6a37a48d04c3
paper_authors.name: Marc'Aurelio Ranzato
paper_title: Towards Robust and Efficient Continual Language Learning
paper_summary: As the application space of language models continues to evolve, a natural
  question to ask is how we can quickly adapt models to new tasks. We approach
  this classic question from a continual learning perspective, in which we aim to
  continue fine-tuning models trained on past tasks on new tasks, with the goal
  of "transferring" relevant knowledge. However, this strategy also runs the risk
  of doing more harm than good, i.e., negative transfer. In this paper, we
  construct a new benchmark of task sequences that target different possible
  transfer scenarios one might face, such as a sequence of tasks with high
  potential of positive transfer, high potential for negative transfer, no
  expected effect, or a mixture of each. An ideal learner should be able to
  maximally exploit information from all tasks that have any potential for
  positive transfer, while also avoiding the negative effects of any distracting
  tasks that may confuse it. We then propose a simple, yet effective, learner
  that satisfies many of our desiderata simply by leveraging a selective strategy
  for initializing new models from past task checkpoints. Still, limitations
  remain, and we hope this benchmark can help the community to further build and
  analyze such learners.
paper_ai_summary: A benchmark of task sequences for continual learning evaluates transfer scenarios, and a selective initialization strategy for new models is proposed to mitigate negative transfer.
paper_ai_keywords: continual learning fine-tuning positive transfer negative transfer task sequences selective strategy task checkpoints