title: CamemBERT 2.0: A Smarter French Language Model Aged to Perfection
summary: French language models, such as CamemBERT, have been widely adopted across
  industries for natural language processing (NLP) tasks, with models like
  CamemBERT seeing over 4 million downloads per month. However, these models face
  challenges due to temporal concept drift, where outdated training data leads to
  a decline in performance, especially when encountering new topics and
  terminology. This issue emphasizes the need for updated models that reflect
  current linguistic trends. In this paper, we introduce two new versions of the
  CamemBERT base model-CamemBERTav2 and CamemBERTv2-designed to address these
  challenges. CamemBERTav2 is based on the DeBERTaV3 architecture and makes use
  of the Replaced Token Detection (RTD) objective for better contextual
  understanding, while CamemBERTv2 is built on RoBERTa, which uses the Masked
  Language Modeling (MLM) objective. Both models are trained on a significantly
  larger and more recent dataset with longer context length and an updated
  tokenizer that enhances tokenization performance for French. We evaluate the
  performance of these models on both general-domain NLP tasks and
  domain-specific applications, such as medical field tasks, demonstrating their
  versatility and effectiveness across a range of use cases. Our results show
  that these updated models vastly outperform their predecessors, making them
  valuable tools for modern NLP systems. All our new models, as well as
  intermediate checkpoints, are made openly available on Huggingface.
paper_authors._id: 6735b1a8bee2d791dc7f9367
paper_authors.name: Wissam Antoun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:21:48.082Z
paper_authors._id: 605350699008a64d7b43b78d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1616431735085-605350699008a64d7b43b78d.jpeg
paper_authors.fullname: Wissam Antoun
paper_authors.type: user
paper_authors.user: wissamantoun
paper_authors._id: 6735b1a8bee2d791dc7f9368
paper_authors.name: Francis Kulumba
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:21:50.051Z
paper_authors._id: 622a058138f0b01c1c2b33c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/622a058138f0b01c1c2b33c9/fZ2T_BJU9gbXGuxgbZ_OI.jpeg
paper_authors.fullname: Francis Kulumba
paper_authors.type: user
paper_authors.user: Madjakul
paper_authors._id: 6735b1a8bee2d791dc7f9369
paper_authors.name: Rian Touchent
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:21:14.167Z
paper_authors._id: 62a9b0acf6708cb85014f9dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a9b0acf6708cb85014f9dc/Sem1qcBt1lJjFEPK-xz4_.jpeg
paper_authors.fullname: Rian Touchent
paper_authors.type: user
paper_authors.user: rntc
paper_authors._id: 6735b1a8bee2d791dc7f936a
paper_authors.name: Éric de la Clergerie
paper_authors._id: 6735b1a8bee2d791dc7f936b
paper_authors.name: Benoît Sagot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:19:04.513Z
paper_authors._id: 602ba2a739515f8d31237967
paper_authors.avatarUrl: /avatars/ef534712ca682bf74ec7eef17d3d1b5f.svg
paper_authors.fullname: Benoît Sagot
paper_authors.type: user
paper_authors.user: sagot
paper_authors._id: 6735b1a8bee2d791dc7f936c
paper_authors.name: Djamé Seddah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:19:14.247Z
paper_authors._id: 638b6bf3bb4068fd07c2b261
paper_authors.avatarUrl: /avatars/3aefd8879773a7d1afabfd02d2a47b1d.svg
paper_authors.fullname: Djamé Seddah
paper_authors.type: user
paper_authors.user: Djame
paper_title: CamemBERT 2.0: A Smarter French Language Model Aged to Perfection
paper_summary: French language models, such as CamemBERT, have been widely adopted across
  industries for natural language processing (NLP) tasks, with models like
  CamemBERT seeing over 4 million downloads per month. However, these models face
  challenges due to temporal concept drift, where outdated training data leads to
  a decline in performance, especially when encountering new topics and
  terminology. This issue emphasizes the need for updated models that reflect
  current linguistic trends. In this paper, we introduce two new versions of the
  CamemBERT base model-CamemBERTav2 and CamemBERTv2-designed to address these
  challenges. CamemBERTav2 is based on the DeBERTaV3 architecture and makes use
  of the Replaced Token Detection (RTD) objective for better contextual
  understanding, while CamemBERTv2 is built on RoBERTa, which uses the Masked
  Language Modeling (MLM) objective. Both models are trained on a significantly
  larger and more recent dataset with longer context length and an updated
  tokenizer that enhances tokenization performance for French. We evaluate the
  performance of these models on both general-domain NLP tasks and
  domain-specific applications, such as medical field tasks, demonstrating their
  versatility and effectiveness across a range of use cases. Our results show
  that these updated models vastly outperform their predecessors, making them
  valuable tools for modern NLP systems. All our new models, as well as
  intermediate checkpoints, are made openly available on Huggingface.
paper_ai_summary: New versions of CamemBERT, CamemBERTav2 and CamemBERTv2, address temporal concept drift using DeBERTaV3 and RoBERTa architectures, respectively, outperforming their predecessors across various NLP tasks.
paper_ai_keywords: CamemBERT DeBERTaV3 Replaced Token Detection RTD RoBERTa Masked Language Modeling MLM context length tokenizer Huggingface