CamemBERT 2.0: A Smarter French Language Model Aged to Perfection
French language models, such as CamemBERT, have been widely adopted across
industries for natural language processing (NLP) tasks, with models like
CamemBERT seeing over 4 million downloads per month. However, these models face
challenges due to temporal concept drift, where outdated training data leads to
a decline in performance, especially when encountering new topics and
terminology. This issue emphasizes the need for updated models that reflect
current linguistic trends. In this paper, we introduce two new versions of the
CamemBERT base model-CamemBERTav2 and CamemBERTv2-designed to address these
challenges. CamemBERTav2 is based on the DeBERTaV3 architecture and makes use
of the Replaced Token Detection (RTD) objective for better contextual
understanding, while CamemBERTv2 is built on RoBERTa, which uses the Masked
Language Modeling (MLM) objective. Both models are trained on a significantly
larger and more recent dataset with longer context length and an updated
tokenizer that enhances tokenization performance for French. We evaluate the
performance of these models on both general-domain NLP tasks and
domain-specific applications, such as medical field tasks, demonstrating their
versatility and effectiveness across a range of use cases. Our results show
that these updated models vastly outperform their predecessors, making them
valuable tools for modern NLP systems. All our new models, as well as
intermediate checkpoints, are made openly available on Huggingface.
6735b1a8bee2d791dc7f9367
Wissam Antoun
claimed_verified
2024-11-15T09:21:48.082Z
605350699008a64d7b43b78d
https://cdn-avatars.huggingface.co/v1/production/uploads/1616431735085-605350699008a64d7b43b78d.jpeg
Wissam Antoun
user
wissamantoun
6735b1a8bee2d791dc7f9368
Francis Kulumba
claimed_verified
2024-11-15T09:21:50.051Z
622a058138f0b01c1c2b33c9
https://cdn-avatars.huggingface.co/v1/production/uploads/622a058138f0b01c1c2b33c9/fZ2T_BJU9gbXGuxgbZ_OI.jpeg
Francis Kulumba
user
Madjakul
6735b1a8bee2d791dc7f9369
Rian Touchent
claimed_verified
2024-11-15T09:21:14.167Z
62a9b0acf6708cb85014f9dc
https://cdn-avatars.huggingface.co/v1/production/uploads/62a9b0acf6708cb85014f9dc/Sem1qcBt1lJjFEPK-xz4_.jpeg
Rian Touchent
user
rntc
6735b1a8bee2d791dc7f936a
Éric de la Clergerie
6735b1a8bee2d791dc7f936b
Benoît Sagot
admin_assigned
2024-11-15T14:19:04.513Z
602ba2a739515f8d31237967
/avatars/ef534712ca682bf74ec7eef17d3d1b5f.svg
Benoît Sagot
user
sagot
6735b1a8bee2d791dc7f936c
Djamé Seddah
admin_assigned
2024-11-15T14:19:14.247Z
638b6bf3bb4068fd07c2b261
/avatars/3aefd8879773a7d1afabfd02d2a47b1d.svg
Djamé Seddah
user
Djame
CamemBERT 2.0: A Smarter French Language Model Aged to Perfection
French language models, such as CamemBERT, have been widely adopted across
industries for natural language processing (NLP) tasks, with models like
CamemBERT seeing over 4 million downloads per month. However, these models face
challenges due to temporal concept drift, where outdated training data leads to
a decline in performance, especially when encountering new topics and
terminology. This issue emphasizes the need for updated models that reflect
current linguistic trends. In this paper, we introduce two new versions of the
CamemBERT base model-CamemBERTav2 and CamemBERTv2-designed to address these
challenges. CamemBERTav2 is based on the DeBERTaV3 architecture and makes use
of the Replaced Token Detection (RTD) objective for better contextual
understanding, while CamemBERTv2 is built on RoBERTa, which uses the Masked
Language Modeling (MLM) objective. Both models are trained on a significantly
larger and more recent dataset with longer context length and an updated
tokenizer that enhances tokenization performance for French. We evaluate the
performance of these models on both general-domain NLP tasks and
domain-specific applications, such as medical field tasks, demonstrating their
versatility and effectiveness across a range of use cases. Our results show
that these updated models vastly outperform their predecessors, making them
valuable tools for modern NLP systems. All our new models, as well as
intermediate checkpoints, are made openly available on Huggingface.
New versions of CamemBERT, CamemBERTav2 and CamemBERTv2, address temporal concept drift using DeBERTaV3 and RoBERTa architectures, respectively, outperforming their predecessors across various NLP tasks.
CamemBERT DeBERTaV3 Replaced Token Detection RTD RoBERTa Masked Language Modeling MLM context length tokenizer Huggingface