NeoBERT: A Next-Generation BERT
Recent innovations in architecture, pre-training, and fine-tuning have led to
the remarkable in-context learning and reasoning abilities of large
auto-regressive language models such as LLaMA and DeepSeek. In contrast,
encoders like BERT and RoBERTa have not seen the same level of progress despite
being foundational for many downstream NLP applications. To bridge this gap, we
introduce NeoBERT, a next-generation encoder that redefines the capabilities of
bidirectional models by integrating state-of-the-art advancements in
architecture, modern data, and optimized pre-training methodologies. NeoBERT is
designed for seamless adoption: it serves as a plug-and-play replacement for
existing base models, relies on an optimal depth-to-width ratio, and leverages
an extended context length of 4,096 tokens. Despite its compact 250M parameter
footprint, it achieves state-of-the-art results on the massive MTEB benchmark,
outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under
identical fine-tuning conditions. In addition, we rigorously evaluate the
impact of each modification on GLUE and design a uniform fine-tuning and
evaluation framework for MTEB. We release all code, data, checkpoints, and
training scripts to accelerate research and real-world adoption.
67c13aa6a43d7939d60eb02e
Lola Le Breton
admin_assigned
2025-02-28T12:35:52.732Z
6512e961332b85e7cf8c1431
/avatars/d4bdb9670166112dcb36753bc1823b28.svg
Lola Le Breton
user
Lolalb
67c13aa6a43d7939d60eb02f
Quentin Fournier
claimed_verified
2025-03-25T08:21:39.314Z
63d2ca0ac20657d24348cf42
https://cdn-avatars.huggingface.co/v1/production/uploads/63d2ca0ac20657d24348cf42/foR1vMVCzczq2S-NOJQ8A.jpeg
Quentin Fournier
user
qfournier
67c13aa6a43d7939d60eb030
Mariam El Mezouar
admin_assigned
2025-02-28T12:36:28.502Z
6504c139eac45ee2e4d36893
/avatars/cc3d65f558988ab885aa0357f6e2d29d.svg
Mariam El Mezouar
user
mariamelm
67c13aa6a43d7939d60eb031
Sarath Chandar
admin_assigned
2025-02-28T12:36:22.663Z
66fabb66c2bf89d75e8cdd4d
/avatars/c40f55a77e2fb34ba38a79f04df82893.svg
Sarath Chandar
user
apsarath
NeoBERT: A Next-Generation BERT
Recent innovations in architecture, pre-training, and fine-tuning have led to
the remarkable in-context learning and reasoning abilities of large
auto-regressive language models such as LLaMA and DeepSeek. In contrast,
encoders like BERT and RoBERTa have not seen the same level of progress despite
being foundational for many downstream NLP applications. To bridge this gap, we
introduce NeoBERT, a next-generation encoder that redefines the capabilities of
bidirectional models by integrating state-of-the-art advancements in
architecture, modern data, and optimized pre-training methodologies. NeoBERT is
designed for seamless adoption: it serves as a plug-and-play replacement for
existing base models, relies on an optimal depth-to-width ratio, and leverages
an extended context length of 4,096 tokens. Despite its compact 250M parameter
footprint, it achieves state-of-the-art results on the massive MTEB benchmark,
outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under
identical fine-tuning conditions. In addition, we rigorously evaluate the
impact of each modification on GLUE and design a uniform fine-tuning and
evaluation framework for MTEB. We release all code, data, checkpoints, and
training scripts to accelerate research and real-world adoption.
NeoBERT is a next-generation encoder that integrates modern advancements to surpass performance of BERT and RoBERTa on MTEB with a compact parameter footprint.
NeoBERT state-of-the-art advancements bidirectional models depth-to-width ratio extended context length MTEB benchmark BERT large RoBERTa large NomicBERT ModernBERT GLUE uniform fine-tuning evaluation framework