Vatolin Alexey

vatolinalex

AI & ML interests

None yet

Recent Activity

liked a model 24 days ago
EuroBERT/EuroBERT-210m
reacted to tomaarsen's post with ā¤ļø 24 days ago
An assembly of 18 European companies, labs, and universities have banded together to launch šŸ‡ŖšŸ‡ŗ EuroBERT! It's a state-of-the-art multilingual encoder for 15 European languages, designed to be finetuned for retrieval, classification, etc. šŸ‡ŖšŸ‡ŗ 15 Languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi 3ļøāƒ£ 3 model sizes: 210M, 610M, and 2.1B parameters - very very useful sizes in my opinion āž”ļø Sequence length of 8192 tokens! Nice to see these higher sequence lengths for encoders becoming more common. āš™ļø Architecture based on Llama, but with bi-directional (non-causal) attention to turn it into an encoder. Flash Attention 2 is supported. šŸ”„ A new Pareto frontier (stronger *and* smaller) for multilingual encoder models šŸ“Š Evaluated against mDeBERTa, mGTE, XLM-RoBERTa for Retrieval, Classification, and Regression (after finetuning for each task separately): EuroBERT punches way above its weight. šŸ“ Detailed paper with all details, incl. data: FineWeb for English and CulturaX for multilingual data, The Stack v2 and Proof-Pile-2 for code. Check out the release blogpost here: https://huggingface.co/blog/EuroBERT/release * https://huggingface.co/EuroBERT/EuroBERT-210m * https://huggingface.co/EuroBERT/EuroBERT-610m * https://huggingface.co/EuroBERT/EuroBERT-2.1B The next step is for researchers to build upon the 3 EuroBERT base models and publish strong retrieval, zero-shot classification, etc. models for all to use. I'm very much looking forward to it!
View all activity

Organizations

Massive Text Embedding Benchmark's profile picture

vatolinalex's activity

reacted to tomaarsen's post with ā¤ļø 24 days ago
view post
Post
6611
An assembly of 18 European companies, labs, and universities have banded together to launch šŸ‡ŖšŸ‡ŗ EuroBERT! It's a state-of-the-art multilingual encoder for 15 European languages, designed to be finetuned for retrieval, classification, etc.

šŸ‡ŖšŸ‡ŗ 15 Languages: English, French, German, Spanish, Chinese, Italian, Russian, Polish, Portuguese, Japanese, Vietnamese, Dutch, Arabic, Turkish, Hindi
3ļøāƒ£ 3 model sizes: 210M, 610M, and 2.1B parameters - very very useful sizes in my opinion
āž”ļø Sequence length of 8192 tokens! Nice to see these higher sequence lengths for encoders becoming more common.
āš™ļø Architecture based on Llama, but with bi-directional (non-causal) attention to turn it into an encoder. Flash Attention 2 is supported.
šŸ”„ A new Pareto frontier (stronger *and* smaller) for multilingual encoder models
šŸ“Š Evaluated against mDeBERTa, mGTE, XLM-RoBERTa for Retrieval, Classification, and Regression (after finetuning for each task separately): EuroBERT punches way above its weight.
šŸ“ Detailed paper with all details, incl. data: FineWeb for English and CulturaX for multilingual data, The Stack v2 and Proof-Pile-2 for code.

Check out the release blogpost here: https://huggingface.co/blog/EuroBERT/release
* EuroBERT/EuroBERT-210m
* EuroBERT/EuroBERT-610m
* EuroBERT/EuroBERT-2.1B

The next step is for researchers to build upon the 3 EuroBERT base models and publish strong retrieval, zero-shot classification, etc. models for all to use. I'm very much looking forward to it!
  • 1 reply
Ā·
New activity in Vikhrmodels/habr_qa_sbs 4 months ago

Fix dataset reading error

#1 opened 4 months ago by
vatolinalex