title: Improving Language Plasticity via Pretraining with Active Forgetting
summary: Pretrained language models (PLMs) are today the primary model for natural
  language processing. Despite their impressive downstream performance, it can be
  difficult to apply PLMs to new languages, a barrier to making their
  capabilities universally accessible. While prior work has shown it possible to
  address this issue by learning a new embedding layer for the new language,
  doing so is both data and compute inefficient. We propose to use an active
  forgetting mechanism during pretraining, as a simple way of creating PLMs that
  can quickly adapt to new languages. Concretely, by resetting the embedding
  layer every K updates during pretraining, we encourage the PLM to improve its
  ability of learning new embeddings within a limited number of updates, similar
  to a meta-learning effect. Experiments with RoBERTa show that models pretrained
  with our forgetting mechanism not only demonstrate faster convergence during
  language adaptation but also outperform standard ones in a low-data regime,
  particularly for languages that are distant from English.
paper_authors._id: 64a387a702617d262f8a7cff
paper_authors.name: Yihong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:37:13.413Z
paper_authors._id: 6674243633e45ebfcdc8be3b
paper_authors.avatarUrl: /avatars/4e87009e3576c9351d70bc8223538b9e.svg
paper_authors.fullname: yihong
paper_authors.type: user
paper_authors.user: yihong-chen
paper_authors._id: 64a387a702617d262f8a7d00
paper_authors.name: Kelly Marchisio
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-03T22:54:39.338Z
paper_authors._id: 66356b1e2fcf64c1548a1233
paper_authors.avatarUrl: /avatars/9a1fab7da6cd16e8bf2d104c399ae34f.svg
paper_authors.fullname: Kelly Marchisio
paper_authors.type: user
paper_authors.user: kellycohere
paper_authors._id: 64a387a702617d262f8a7d01
paper_authors.name: Roberta Raileanu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-07-04T02:44:56.019Z
paper_authors._id: 633e94793a17ab61de8e2b9c
paper_authors.avatarUrl: /avatars/5f2f58ddeed211393660ada6b135f0d5.svg
paper_authors.fullname: Roberta Raileanu
paper_authors.type: user
paper_authors.user: rraileanu
paper_authors._id: 64a387a702617d262f8a7d02
paper_authors.name: David Ifeoluwa Adelani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T19:16:57.055Z
paper_authors._id: 5fcc1929563427b03e9af259
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620642852901-5fcc1929563427b03e9af259.jpeg
paper_authors.fullname: David Adelani
paper_authors.type: user
paper_authors.user: Davlan
paper_authors._id: 64a387a702617d262f8a7d03
paper_authors.name: Pontus Stenetor
paper_authors._id: 64a387a702617d262f8a7d04
paper_authors.name: Sebastian Riedel
paper_authors._id: 64a387a702617d262f8a7d05
paper_authors.name: Mikel Artetx
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-09-18T08:38:05.072Z
paper_authors._id: 65080c544a8839a8bda2cfbd
paper_authors.avatarUrl: /avatars/8684a50e82bd21e14c9f32c3a443760d.svg
paper_authors.fullname: Mikel Artetxe
paper_authors.type: user
paper_authors.user: artetxem
paper_title: Improving Language Plasticity via Pretraining with Active Forgetting
paper_summary: Pretrained language models (PLMs) are today the primary model for natural
  language processing. Despite their impressive downstream performance, it can be
  difficult to apply PLMs to new languages, a barrier to making their
  capabilities universally accessible. While prior work has shown it possible to
  address this issue by learning a new embedding layer for the new language,
  doing so is both data and compute inefficient. We propose to use an active
  forgetting mechanism during pretraining, as a simple way of creating PLMs that
  can quickly adapt to new languages. Concretely, by resetting the embedding
  layer every K updates during pretraining, we encourage the PLM to improve its
  ability of learning new embeddings within a limited number of updates, similar
  to a meta-learning effect. Experiments with RoBERTa show that models pretrained
  with our forgetting mechanism not only demonstrate faster convergence during
  language adaptation but also outperform standard ones in a low-data regime,
  particularly for languages that are distant from English.
paper_ai_keywords: pretrained language models natural language processing embedding layer active forgetting mechanism meta-learning language adaptation low-data regime