title: EuroBERT: Scaling Multilingual Encoders for European Languages
summary: General-purpose multilingual vector representations, used in retrieval,
  regression and classification, are traditionally obtained from bidirectional
  encoder models. Despite their wide applicability, encoders have been recently
  overshadowed by advances in generative decoder-only models. However, many
  innovations driving this progress are not inherently tied to decoders. In this
  paper, we revisit the development of multilingual encoders through the lens of
  these advances, and introduce EuroBERT, a family of multilingual encoders
  covering European and widely spoken global languages. Our models outperform
  existing alternatives across a diverse range of tasks, spanning multilingual
  capabilities, mathematics, and coding, and natively supporting sequences of up
  to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering
  insights into our dataset composition and training pipeline. We publicly
  release the EuroBERT models, including intermediate training checkpoints,
  together with our training framework.
paper_authors._id: 67ce9626e5cdfda52b9e8839
paper_authors.name: Nicolas Boizard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:42:06.860Z
paper_authors._id: 62be186a5f59ff2320e6e32b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62be186a5f59ff2320e6e32b/W_emoC2uItM-MJZyCfIKI.png
paper_authors.fullname: Nicolas-BZRD
paper_authors.type: user
paper_authors.user: Nicolas-BZRD
paper_authors._id: 67ce9626e5cdfda52b9e883a
paper_authors.name: Hippolyte Gisserot-Boukhlef
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:42:13.176Z
paper_authors._id: 65fa95405355a52c784633fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65fa95405355a52c784633fc/rSfBUHPa7eSAsLd8DuOq4.png
paper_authors.fullname: Hippolyte Gisserot-Boukhlef
paper_authors.type: user
paper_authors.user: hgissbkh
paper_authors._id: 67ce9626e5cdfda52b9e883b
paper_authors.name: Duarte M. Alves
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:42:23.055Z
paper_authors._id: 64132452d8a418df415a6ded
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64132452d8a418df415a6ded/qkjL5G89uldHUXlCI3n4f.jpeg
paper_authors.fullname: Duarte Alves
paper_authors.type: user
paper_authors.user: DuarteMRAlves
paper_authors._id: 67ce9626e5cdfda52b9e883c
paper_authors.name: André Martins
paper_authors._id: 67ce9626e5cdfda52b9e883d
paper_authors.name: Ayoub Hammal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:42:42.527Z
paper_authors._id: 63937b399762cdd66be2a32f
paper_authors.avatarUrl: /avatars/7aefd888a3c54673d5881dcef61f771b.svg
paper_authors.fullname: Ayoub Hammal
paper_authors.type: user
paper_authors.user: ayoubhammal
paper_authors._id: 67ce9626e5cdfda52b9e883e
paper_authors.name: Caio Corro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:42:48.603Z
paper_authors._id: 677bedd522ca8585ede98470
paper_authors.avatarUrl: /avatars/54bca410c446610f02aca55918c74518.svg
paper_authors.fullname: Caio Corro
paper_authors.type: user
paper_authors.user: caiocorro
paper_authors._id: 67ce9626e5cdfda52b9e883f
paper_authors.name: Céline Hudelot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:51:41.273Z
paper_authors._id: 61efea03a57920a251ec19b8
paper_authors.avatarUrl: /avatars/f47c8e3cb17a2bf7d43f2c152bb86885.svg
paper_authors.fullname: Celine Hudelot
paper_authors.type: user
paper_authors.user: CelineH
paper_authors._id: 67ce9626e5cdfda52b9e8840
paper_authors.name: Emmanuel Malherbe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:51:47.996Z
paper_authors._id: 66f2d6a684a241caac8e16dc
paper_authors.avatarUrl: /avatars/81acb87c2b07bea938251b40a2139911.svg
paper_authors.fullname: Emmanuel Malherbe
paper_authors.type: user
paper_authors.user: emmanuelmalherbe
paper_authors._id: 67ce9626e5cdfda52b9e8841
paper_authors.name: Etienne Malaboeuf
paper_authors._id: 67ce9626e5cdfda52b9e8842
paper_authors.name: Fanny Jourdan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:52:09.223Z
paper_authors._id: 6708db59caf70ddea8e1355d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6708db59caf70ddea8e1355d/C6T16AdpqoeWCk7Gg9wSH.jpeg
paper_authors.fullname: Fanny Jourdan
paper_authors.type: user
paper_authors.user: Fannyjrd
paper_authors._id: 67ce9626e5cdfda52b9e8843
paper_authors.name: Gabriel Hautreux
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:52:15.512Z
paper_authors._id: 67cafedda972115e89972cd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/P_xComqG9IttvluN-6tyB.png
paper_authors.fullname: Gabriel Hautreux
paper_authors.type: user
paper_authors.user: GabrielHau
paper_authors._id: 67ce9626e5cdfda52b9e8844
paper_authors.name: João Alves
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:52:22.630Z
paper_authors._id: 6772bde5c997eeb5550e80ea
paper_authors.avatarUrl: /avatars/8134a4d9330317e748dc7b33e1bb25f6.svg
paper_authors.fullname: João Alves
paper_authors.type: user
paper_authors.user: albusonrails
paper_authors._id: 67ce9626e5cdfda52b9e8845
paper_authors.name: Kevin El-Haddad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:52:31.191Z
paper_authors._id: 66e2c22d7cc3edd60d725267
paper_authors.avatarUrl: /avatars/b217c5708c7dba8b1c220f37984ccc1e.svg
paper_authors.fullname: Kevin El Haddad
paper_authors.type: user
paper_authors.user: kelhad
paper_authors._id: 67ce9626e5cdfda52b9e8846
paper_authors.name: Manuel Faysse
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:52:38.114Z
paper_authors._id: 60f2e021adf471cbdf8bb660
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
paper_authors.fullname: Manuel Faysse
paper_authors.type: user
paper_authors.user: manu
paper_authors._id: 67ce9626e5cdfda52b9e8847
paper_authors.name: Maxime Peyrard
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:52:44.389Z
paper_authors._id: 6369394dd322a76e1ea4bdf6
paper_authors.avatarUrl: /avatars/a4e5ab0167025fbbfc970d54630ce754.svg
paper_authors.fullname: Maxime Peyrard
paper_authors.type: user
paper_authors.user: peyrardm
paper_authors._id: 67ce9626e5cdfda52b9e8848
paper_authors.name: Nuno M. Guerreiro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:52:54.367Z
paper_authors._id: 67b622d2df3a86fbca306c43
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lNlshrl56oaKslArMzSzj.png
paper_authors.fullname: Nuno  Guerreiro
paper_authors.type: user
paper_authors.user: nunogj
paper_authors._id: 67ce9626e5cdfda52b9e8849
paper_authors.name: Patrick Fernandes
paper_authors._id: 67ce9626e5cdfda52b9e884a
paper_authors.name: Ricardo Rei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:18:40.390Z
paper_authors._id: 60d9a9791fa5d458da77754d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624887555363-60d9a9791fa5d458da77754d.jpeg
paper_authors.fullname: Ricardo Costa Dias Rei
paper_authors.type: user
paper_authors.user: RicardoRei
paper_authors._id: 67ce9626e5cdfda52b9e884b
paper_authors.name: Pierre Colombo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-10T09:41:26.353Z
paper_authors._id: 644a900e3a619fe72b14af0f
paper_authors.avatarUrl: /avatars/e2d5dac3d92757ed48e37e126a3464a3.svg
paper_authors.fullname: Colombo
paper_authors.type: user
paper_authors.user: PierreColombo
paper_title: EuroBERT: Scaling Multilingual Encoders for European Languages
paper_summary: General-purpose multilingual vector representations, used in retrieval,
  regression and classification, are traditionally obtained from bidirectional
  encoder models. Despite their wide applicability, encoders have been recently
  overshadowed by advances in generative decoder-only models. However, many
  innovations driving this progress are not inherently tied to decoders. In this
  paper, we revisit the development of multilingual encoders through the lens of
  these advances, and introduce EuroBERT, a family of multilingual encoders
  covering European and widely spoken global languages. Our models outperform
  existing alternatives across a diverse range of tasks, spanning multilingual
  capabilities, mathematics, and coding, and natively supporting sequences of up
  to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering
  insights into our dataset composition and training pipeline. We publicly
  release the EuroBERT models, including intermediate training checkpoints,
  together with our training framework.
paper_ai_keywords: bidirectional encoder models generative decoder-only models multilingual encoders EuroBERT multilingual capabilities sequences of up to 8,192 tokens intermediate training checkpoints training framework