An assembly of 18 European companies, labs, and universities have banded together to launch 🇪🇺 EuroBERT! It's a state-of-the-art multilingual encoder for 15 European languages, designed to be finetuned for retrieval, classification, etc. 
text
An assembly of 18 European companies, labs, and universities have banded together to launch 🇪🇺 EuroBERT! It's a state-of-the-art multilingual encoder for 15 European languages, designed to be finetuned for retrieval, classification, etc. 
new_line
new_line
🇪🇺 15 Languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi
text
🇪🇺 15 Languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi
new_line
3️⃣ 3 model sizes: 210M, 610M, and 2.1B parameters - very very useful sizes in my opinion
text
3️⃣ 3 model sizes: 210M, 610M, and 2.1B parameters - very very useful sizes in my opinion
new_line
➡️ Sequence length of 8192 tokens! Nice to see these higher sequence lengths for encoders becoming more common.
text
➡️ Sequence length of 8192 tokens! Nice to see these higher sequence lengths for encoders becoming more common.
new_line
⚙️ Architecture based on Llama, but with bi-directional (non-causal) attention to turn it into an encoder. Flash Attention 2 is supported.
text
⚙️ Architecture based on Llama, but with bi-directional (non-causal) attention to turn it into an encoder. Flash Attention 2 is supported.
new_line
🔥 A new Pareto frontier (stronger *and* smaller) for multilingual encoder models
text
🔥 A new Pareto frontier (stronger *and* smaller) for multilingual encoder models
new_line
📊 Evaluated against mDeBERTa, mGTE, XLM-RoBERTa for Retrieval, Classification, and Regression (after finetuning for each task separately): EuroBERT punches way above its weight.
text
📊 Evaluated against mDeBERTa, mGTE, XLM-RoBERTa for Retrieval, Classification, and Regression (after finetuning for each task separately): EuroBERT punches way above its weight.
new_line
📝 Detailed paper with all details, incl. data: FineWeb for English and CulturaX for multilingual data, The Stack v2 and Proof-Pile-2 for code.
text
📝 Detailed paper with all details, incl. data: FineWeb for English and CulturaX for multilingual data, The Stack v2 and Proof-Pile-2 for code.
new_line
new_line
Check out the release blogpost here: 
text
Check out the release blogpost here: 
https://huggingface.co/blog/EuroBERT/release
https://huggingface.co/blog/EuroBERT/release
link
new_line
* 
text
* 
https://huggingface.co/EuroBERT/EuroBERT-210m
EuroBERT/EuroBERT-210m
model
resource
https://huggingface.co/EuroBERT/EuroBERT-210m
new_line
* 
text
* 
https://huggingface.co/EuroBERT/EuroBERT-610m
EuroBERT/EuroBERT-610m
model
resource
https://huggingface.co/EuroBERT/EuroBERT-610m
new_line
* 
text
* 
https://huggingface.co/EuroBERT/EuroBERT-2.1B
EuroBERT/EuroBERT-2.1B
model
resource
https://huggingface.co/EuroBERT/EuroBERT-2.1B
new_line
new_line
The next step is for researchers to build upon the 3 EuroBERT base models and publish strong retrieval, zero-shot classification, etc. models for all to use. I'm very much looking forward to it!
text
The next step is for researchers to build upon the 3 EuroBERT base models and publish strong retrieval, zero-shot classification, etc. models for all to use. I'm very much looking forward to it!
An assembly of 18 European companies, labs, and universities have banded together to launch 🇪🇺 EuroBERT! It's a state-of-the-art multilingual encoder for 15 European languages, designed to be finetuned for retrieval, classification, etc. 

🇪🇺 15 Languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi
3️⃣ 3 model sizes: 210M, 610M, and 2.1B parameters - very very useful sizes in my opinion
➡️ Sequence length of 8192 tokens! Nice to see these higher sequence lengths for encoders becoming more common.
⚙️ Architecture based on Llama, but with bi-directional (non-causal) attention to turn it into an encoder. Flash Attention 2 is supported.
🔥 A new Pareto frontier (stronger *and* smaller) for multilingual encoder models
📊 Evaluated against mDeBERTa, mGTE, XLM-RoBERTa for Retrieval, Classification, and Regression (after finetuning for each task separately): EuroBERT punches way above its weight.
📝 Detailed paper with all details, incl. data: FineWeb for English and CulturaX for multilingual data, The Stack v2 and Proof-Pile-2 for code.

Check out the release blogpost here: https://huggingface.co/blog/EuroBERT/release
* https://huggingface.co/EuroBERT/EuroBERT-210m
* https://huggingface.co/EuroBERT/EuroBERT-610m
* https://huggingface.co/EuroBERT/EuroBERT-2.1B

The next step is for researchers to build upon the 3 EuroBERT base models and publish strong retrieval, zero-shot classification, etc. models for all to use. I'm very much looking forward to it!
❤️
Nicolas-BZRD MexIvanov cnmoro okasag ritvik77 BrigitteTousi vatolinalex ysenarath eaddario fireblade2534 agentlans RedAndFoxy akahana robb-0 Rexschwert tolgacangoz Ainonake samheym Fishtiks Jaume dantezxcd
🔥
CharlesMoslonka John6666 flozi00 DeathGodlike Rexschwert Ameeeee dantezxcd
👍
Yehor
🚀
DeathGodlike
🧠
Rexschwert
663a18c876e6d5b98f3cc94d
/avatars/faa2a47dbdd3bce6d47e0fbed6170627.svg
Alan Tseng
agentlans
user