title: Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for
    Fast, Memory Efficient, and Long Context Finetuning and Inference
summary: Encoder-only transformer models such as BERT offer a great performance-size
  tradeoff for retrieval and classification tasks with respect to larger
  decoder-only models. Despite being the workhorse of numerous production
  pipelines, there have been limited Pareto improvements to BERT since its
  release. In this paper, we introduce ModernBERT, bringing modern model
  optimizations to encoder-only models and representing a major Pareto
  improvement over older encoders. Trained on 2 trillion tokens with a native
  8192 sequence length, ModernBERT models exhibit state-of-the-art results on a
  large pool of evaluations encompassing diverse classification tasks and both
  single and multi-vector retrieval on different domains (including code). In
  addition to strong downstream performance, ModernBERT is also the most speed
  and memory efficient encoder and is designed for inference on common GPUs.
paper_authors._id: 67638c8ed63e4b348e8a5070
paper_authors.name: Benjamin Warner
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:37.192Z
paper_authors._id: 61915ad7913429273972a602
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649175349873-61915ad7913429273972a602.jpeg
paper_authors.fullname: Benjamin Warner
paper_authors.type: user
paper_authors.user: bwarner
paper_authors._id: 67638c8ed63e4b348e8a5071
paper_authors.name: Antoine Chaffin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:27.164Z
paper_authors._id: 609bbe2f4932693ca2009d6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620819560688-609bbe2f4932693ca2009d6a.jpeg
paper_authors.fullname: Antoine Chaffin
paper_authors.type: user
paper_authors.user: NohTow
paper_authors._id: 67638c8ed63e4b348e8a5072
paper_authors.name: Benjamin Clavié
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:19.852Z
paper_authors._id: 5ff60d4352c26e9bc240badd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5ff60d4352c26e9bc240badd/HzoknJibrSasc1ZzU71XA.png
paper_authors.fullname: Benjamin Clavié
paper_authors.type: user
paper_authors.user: bclavie
paper_authors._id: 67638c8ed63e4b348e8a5073
paper_authors.name: Orion Weller
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:24.807Z
paper_authors._id: 6362d9712691058b19de1ba4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
paper_authors.fullname: Orion Weller
paper_authors.type: user
paper_authors.user: orionweller
paper_authors._id: 67638c8ed63e4b348e8a5074
paper_authors.name: Oskar Hallström
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:35:59.025Z
paper_authors._id: 6305ea153aed65d34e9465dc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6305ea153aed65d34e9465dc/g40hxNnctLbfEp6KAC1QG.jpeg
paper_authors.fullname: Oskar Hallström
paper_authors.type: user
paper_authors.user: ohallstrom
paper_authors._id: 67638c8ed63e4b348e8a5075
paper_authors.name: Said Taghadouini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:22.753Z
paper_authors._id: 62cd695e94b9dcedbf1818e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cd695e94b9dcedbf1818e5/qhKhRxdNdbKPIAUfYBvtI.png
paper_authors.fullname: Said Taghadouini
paper_authors.type: user
paper_authors.user: staghado
paper_authors._id: 67638c8ed63e4b348e8a5076
paper_authors.name: Alexis Gallagher
paper_authors._id: 67638c8ed63e4b348e8a5077
paper_authors.name: Raja Biswas
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T13:55:57.854Z
paper_authors._id: 61b839889f7cfeae618e72c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61b839889f7cfeae618e72c9/5kFRCChdqwv7MGM8T_y5v.jpeg
paper_authors.fullname: Raja Biswas
paper_authors.type: user
paper_authors.user: rbiswasfc
paper_authors._id: 67638c8ed63e4b348e8a5078
paper_authors.name: Faisal Ladhak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:36:44.416Z
paper_authors._id: 61ee4f4f53fdc44f4e5e197a
paper_authors.avatarUrl: /avatars/f23f56809805c3e06c580d7420f48494.svg
paper_authors.fullname: Faisal Ladhak
paper_authors.type: user
paper_authors.user: fladhak
paper_authors._id: 67638c8ed63e4b348e8a5079
paper_authors.name: Tom Aarsen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:29.150Z
paper_authors._id: 6317233cc92fd6fee317e030
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png
paper_authors.fullname: Tom Aarsen
paper_authors.type: user
paper_authors.user: tomaarsen
paper_authors._id: 67638c8ed63e4b348e8a507a
paper_authors.name: Nathan Cooper
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-21T15:20:01.022Z
paper_authors._id: 5ebb6509a8e72729bee106a0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633453199694-5ebb6509a8e72729bee106a0.png
paper_authors.fullname: Nathan Cooper
paper_authors.type: user
paper_authors.user: ncoop57
paper_authors._id: 67638c8ed63e4b348e8a507b
paper_authors.name: Griffin Adams
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:36:11.852Z
paper_authors._id: 611693254ef9fdfbf45dc4d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628869402886-noauth.jpeg
paper_authors.fullname: Griffin Adams
paper_authors.type: user
paper_authors.user: griffin
paper_authors._id: 67638c8ed63e4b348e8a507c
paper_authors.name: Jeremy Howard
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:33:38.959Z
paper_authors._id: 623e19ff8972a8c030af5277
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623e19ff8972a8c030af5277/jq9jfT_Sl5B1ZSMHGmBFX.jpeg
paper_authors.fullname: Jeremy Howard
paper_authors.type: user
paper_authors.user: jph00
paper_authors._id: 67638c8ed63e4b348e8a507d
paper_authors.name: Iacopo Poli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:35:51.723Z
paper_authors._id: 62715572ab9243b5d40cbb1d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62715572ab9243b5d40cbb1d/8chtMe-1epj7yVlzVNtrl.png
paper_authors.fullname: Iacopo Poli
paper_authors.type: user
paper_authors.user: iacolippo
paper_title: Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for
    Fast, Memory Efficient, and Long Context Finetuning and Inference
paper_summary: Encoder-only transformer models such as BERT offer a great performance-size
  tradeoff for retrieval and classification tasks with respect to larger
  decoder-only models. Despite being the workhorse of numerous production
  pipelines, there have been limited Pareto improvements to BERT since its
  release. In this paper, we introduce ModernBERT, bringing modern model
  optimizations to encoder-only models and representing a major Pareto
  improvement over older encoders. Trained on 2 trillion tokens with a native
  8192 sequence length, ModernBERT models exhibit state-of-the-art results on a
  large pool of evaluations encompassing diverse classification tasks and both
  single and multi-vector retrieval on different domains (including code). In
  addition to strong downstream performance, ModernBERT is also the most speed
  and memory efficient encoder and is designed for inference on common GPUs.
paper_ai_keywords: encoder-only transformer models BERT decoder-only models ModernBERT model optimizations sequence length state-of-the-art results classification tasks single and multi-vector retrieval code speed and memory efficient