title: NeoBERT: A Next-Generation BERT
summary: Recent innovations in architecture, pre-training, and fine-tuning have led to
  the remarkable in-context learning and reasoning abilities of large
  auto-regressive language models such as LLaMA and DeepSeek. In contrast,
  encoders like BERT and RoBERTa have not seen the same level of progress despite
  being foundational for many downstream NLP applications. To bridge this gap, we
  introduce NeoBERT, a next-generation encoder that redefines the capabilities of
  bidirectional models by integrating state-of-the-art advancements in
  architecture, modern data, and optimized pre-training methodologies. NeoBERT is
  designed for seamless adoption: it serves as a plug-and-play replacement for
  existing base models, relies on an optimal depth-to-width ratio, and leverages
  an extended context length of 4,096 tokens. Despite its compact 250M parameter
  footprint, it achieves state-of-the-art results on the massive MTEB benchmark,
  outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under
  identical fine-tuning conditions. In addition, we rigorously evaluate the
  impact of each modification on GLUE and design a uniform fine-tuning and
  evaluation framework for MTEB. We release all code, data, checkpoints, and
  training scripts to accelerate research and real-world adoption.
paper_authors._id: 67c13aa6a43d7939d60eb02e
paper_authors.name: Lola Le Breton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:35:52.732Z
paper_authors._id: 6512e961332b85e7cf8c1431
paper_authors.avatarUrl: /avatars/d4bdb9670166112dcb36753bc1823b28.svg
paper_authors.fullname: Lola Le Breton
paper_authors.type: user
paper_authors.user: Lolalb
paper_authors._id: 67c13aa6a43d7939d60eb02f
paper_authors.name: Quentin Fournier
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:21:39.314Z
paper_authors._id: 63d2ca0ac20657d24348cf42
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d2ca0ac20657d24348cf42/foR1vMVCzczq2S-NOJQ8A.jpeg
paper_authors.fullname: Quentin Fournier
paper_authors.type: user
paper_authors.user: qfournier
paper_authors._id: 67c13aa6a43d7939d60eb030
paper_authors.name: Mariam El Mezouar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:36:28.502Z
paper_authors._id: 6504c139eac45ee2e4d36893
paper_authors.avatarUrl: /avatars/cc3d65f558988ab885aa0357f6e2d29d.svg
paper_authors.fullname: Mariam El Mezouar
paper_authors.type: user
paper_authors.user: mariamelm
paper_authors._id: 67c13aa6a43d7939d60eb031
paper_authors.name: Sarath Chandar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:36:22.663Z
paper_authors._id: 66fabb66c2bf89d75e8cdd4d
paper_authors.avatarUrl: /avatars/c40f55a77e2fb34ba38a79f04df82893.svg
paper_authors.fullname: Sarath Chandar
paper_authors.type: user
paper_authors.user: apsarath
paper_title: NeoBERT: A Next-Generation BERT
paper_summary: Recent innovations in architecture, pre-training, and fine-tuning have led to
  the remarkable in-context learning and reasoning abilities of large
  auto-regressive language models such as LLaMA and DeepSeek. In contrast,
  encoders like BERT and RoBERTa have not seen the same level of progress despite
  being foundational for many downstream NLP applications. To bridge this gap, we
  introduce NeoBERT, a next-generation encoder that redefines the capabilities of
  bidirectional models by integrating state-of-the-art advancements in
  architecture, modern data, and optimized pre-training methodologies. NeoBERT is
  designed for seamless adoption: it serves as a plug-and-play replacement for
  existing base models, relies on an optimal depth-to-width ratio, and leverages
  an extended context length of 4,096 tokens. Despite its compact 250M parameter
  footprint, it achieves state-of-the-art results on the massive MTEB benchmark,
  outperforming BERT large, RoBERTa large, NomicBERT, and ModernBERT under
  identical fine-tuning conditions. In addition, we rigorously evaluate the
  impact of each modification on GLUE and design a uniform fine-tuning and
  evaluation framework for MTEB. We release all code, data, checkpoints, and
  training scripts to accelerate research and real-world adoption.
paper_ai_summary: NeoBERT is a next-generation encoder that integrates modern advancements to surpass performance of BERT and RoBERTa on MTEB with a compact parameter footprint.
paper_ai_keywords: NeoBERT state-of-the-art advancements bidirectional models depth-to-width ratio extended context length MTEB benchmark BERT large RoBERTa large NomicBERT ModernBERT GLUE uniform fine-tuning evaluation framework