MMTEB: Massive Multilingual Text Embedding Benchmark
Text embeddings are typically evaluated on a limited set of tasks, which are
constrained by language, domain, and task diversity. To address these
limitations and provide a more comprehensive evaluation, we introduce the
Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale,
community-driven expansion of MTEB, covering over 500 quality-controlled
evaluation tasks across 250+ languages. MMTEB includes a diverse set of
challenging, novel tasks such as instruction following, long-document
retrieval, and code retrieval, representing the largest multilingual collection
of evaluation tasks for embedding models to date. Using this collection, we
develop several highly multilingual benchmarks, which we use to evaluate a
representative set of models. We find that while large language models (LLMs)
with billions of parameters can achieve state-of-the-art performance on certain
language subsets and task categories, the best-performing publicly available
model is multilingual-e5-large-instruct with only 560 million parameters. To
facilitate accessibility and reduce computational cost, we introduce a novel
downsampling method based on inter-task correlation, ensuring a diverse
selection while preserving relative model rankings. Furthermore, we optimize
tasks such as retrieval by sampling hard negatives, creating smaller but
effective splits. These optimizations allow us to introduce benchmarks that
drastically reduce computational demands. For instance, our newly introduced
zero-shot English benchmark maintains a ranking order similar to the full-scale
version but at a fraction of the computational cost.
67b6fa9cb544aa153178a60b
Kenneth Enevoldsen
claimed_verified
2025-02-20T15:52:45.751Z
5ff5943752c26e9bc240bada
https://cdn-avatars.huggingface.co/v1/production/uploads/5ff5943752c26e9bc240bada/Exyzf3C_gJ2KdsL4K5_cq.png
Kenneth C. Enevoldsen
user
KennethEnevoldsen
67b6fa9cb544aa153178a60c
Isaac Chung
claimed_verified
2025-02-21T09:59:42.744Z
64cc0e80a257a3212c0c4b24
https://cdn-avatars.huggingface.co/v1/production/uploads/64cc0e80a257a3212c0c4b24/wqs6WZN8-3OQthcnQXgN7.png
Isaac Chung
user
isaacchung
67b6fa9cb544aa153178a60d
Imene Kerboua
claimed_verified
2025-02-21T10:18:18.756Z
62610f8040e04009e81047e9
https://cdn-avatars.huggingface.co/v1/production/uploads/62610f8040e04009e81047e9/iqGTYB7OMmS0jkFEB7cEF.jpeg
Imene Kerboua
user
imenelydiaker
67b6fa9cb544aa153178a60e
Márton Kardos
claimed_verified
2025-02-21T09:59:40.740Z
62696cd3d1ac0cde59280dcf
https://cdn-avatars.huggingface.co/v1/production/uploads/1656940964610-62696cd3d1ac0cde59280dcf.jpeg
Márton Kardos
user
kardosdrur
67b6fa9cb544aa153178a60f
Ashwin Mathur
67b6fa9cb544aa153178a610
David Stap
67b6fa9cb544aa153178a611
Jay Gala
67b6fa9cb544aa153178a612
Wissam Siblini
67b6fa9cb544aa153178a613
Dominik Krzemiński
claimed_verified
2025-02-24T10:58:20.283Z
662cc845616128914a3c9817
/avatars/cd186de00f28bb866abc1ab6c4465663.svg
DomKrz
user
dokato
67b6fa9cb544aa153178a614
Genta Indra Winata
67b6fa9cb544aa153178a615
Saba Sturua
67b6fa9cb544aa153178a616
Saiteja Utpala
67b6fa9cb544aa153178a617
Mathieu Ciancone
67b6fa9cb544aa153178a618
Marion Schaeffer
claimed_verified
2025-02-21T16:39:04.878Z
6527f0678808d80ccff9c230
/avatars/360549c243b11e2d11a75e57b241a5ba.svg
Marion Schaeffer
user
mschaeffer
67b6fa9cb544aa153178a619
Gabriel Sequeira
67b6fa9cb544aa153178a61a
Diganta Misra
67b6fa9cb544aa153178a61b
Shreeya Dhakal
67b6fa9cb544aa153178a61c
Jonathan Rystrøm
67b6fa9cb544aa153178a61d
Roman Solomatin
claimed_verified
2025-02-20T15:52:48.202Z
61af4544d691b3aadd1f62b6
/avatars/7a4067accdd1005f78c3c4adad3ee0a5.svg
Solomatin Roman
user
Samoed
67b6fa9cb544aa153178a61e
Ömer Çağatan
67b6fa9cb544aa153178a61f
Akash Kundu
67b6fa9cb544aa153178a620
Martin Bernstorff
67b6fa9cb544aa153178a621
Shitao Xiao
67b6fa9cb544aa153178a622
Akshita Sukhlecha
67b6fa9cb544aa153178a623
Bhavish Pahwa
claimed_verified
2025-03-16T21:19:26.334Z
623f2f5828672458f74879b3
https://cdn-avatars.huggingface.co/v1/production/uploads/623f2f5828672458f74879b3/E8sWXNEBu1Th1G76XOIqb.jpeg
Bhavish Pahwa
user
bpHigh
67b6fa9cb544aa153178a624
Rafał Poświata
claimed_verified
2025-02-25T09:40:57.852Z
63933543f8b4767ae646e8a1
https://cdn-avatars.huggingface.co/v1/production/uploads/1670591762482-noauth.png
Rafał Poświata
user
rafalposwiata
67b6fa9cb544aa153178a625
Kranthi Kiran GV
67b6fa9cb544aa153178a626
Shawon Ashraf
claimed_verified
2025-04-09T09:48:26.339Z
61b326c20d381ede9f1cf0d1
https://cdn-avatars.huggingface.co/v1/production/uploads/61b326c20d381ede9f1cf0d1/Ckelwn-fKcbA9p3hhedmc.jpeg
Shawon Ashraf
user
shawon
67b6fa9cb544aa153178a627
Daniel Auras
claimed_verified
2025-02-24T09:24:56.683Z
642c54a5b09c70b36de03071
https://cdn-avatars.huggingface.co/v1/production/uploads/642c54a5b09c70b36de03071/EwQyQmust01dgkdRMtYFt.jpeg
rasdani
user
rasdani
67b6fa9cb544aa153178a628
Björn Plüster
67b6fa9cb544aa153178a629
Jan Philipp Harries
67b6fa9cb544aa153178a62a
Loïc Magne
67b6fa9cb544aa153178a62b
Isabelle Mohr
67b6fa9cb544aa153178a62c
Mariya Hendriksen
claimed_verified
2025-02-24T09:23:49.198Z
63960f35339bf68bc775b468
/avatars/d82a914df001b830378246904634b756.svg
Mariya Hendrikse
user
mariya-he
67b6fa9cb544aa153178a62d
Dawei Zhu
claimed_verified
2025-03-25T08:21:44.343Z
64d2fce8129a210e569e0c76
/avatars/a79a832dc3a46ece1b9e542369fc4888.svg
Dawei Zhu
user
dwzhu
67b6fa9cb544aa153178a62e
Hippolyte Gisserot-Boukhlef
claimed_verified
2025-03-24T16:39:54.479Z
65fa95405355a52c784633fc
https://cdn-avatars.huggingface.co/v1/production/uploads/65fa95405355a52c784633fc/rSfBUHPa7eSAsLd8DuOq4.png
Hippolyte Gisserot-Boukhlef
user
hgissbkh
67b6fa9cb544aa153178a62f
Tom Aarsen
claimed_verified
2025-02-21T15:15:34.699Z
6317233cc92fd6fee317e030
https://cdn-avatars.huggingface.co/v1/production/uploads/6317233cc92fd6fee317e030/cJHSvvimr1kqgQfHOjO5n.png
Tom Aarsen
user
tomaarsen
67b6fa9cb544aa153178a630
Jan Kostkan
67b6fa9cb544aa153178a631
Konrad Wojtasik
67b6fa9cb544aa153178a632
Taemin Lee
67b6fa9cb544aa153178a633
Marek Šuppa
67b6fa9cb544aa153178a634
Crystina Zhang
claimed_verified
2025-03-10T08:03:54.291Z
611fb1cbfa8355ed0309de81
https://cdn-avatars.huggingface.co/v1/production/uploads/1665676377085-611fb1cbfa8355ed0309de81.jpeg
Xinyu ZHANG
user
crystina-z
67b6fa9cb544aa153178a635
Roberta Rocca
67b6fa9cb544aa153178a636
Mohammed Hamdy
claimed_verified
2025-02-21T09:59:46.087Z
62645f88c39850dc093d6105
https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
Mohammed Hamdy
user
mmhamdy
67b6fa9cb544aa153178a637
Andrianos Michail
claimed_verified
2025-02-28T13:07:57.365Z
60e1bc418479fac0bd1daa0e
/avatars/61f4f8ac0714aae7d5cbb7d4e1038020.svg
Andrianos Michail
user
Andrianos
67b6fa9cb544aa153178a638
John Yang
claimed_verified
2025-05-07T08:10:09.198Z
677c2e5032582d07139f8af1
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ZxdbXnPAFq_O20FAZuD4v.png
John
user
john-b-yang
67b6fa9cb544aa153178a639
Manuel Faysse
claimed_verified
2025-03-10T08:03:51.735Z
60f2e021adf471cbdf8bb660
https://cdn-avatars.huggingface.co/v1/production/uploads/1654090481550-60f2e021adf471cbdf8bb660.jpeg
Manuel Faysse
user
manu
67b6fa9cb544aa153178a63a
Aleksei Vatolin
claimed_verified
2025-02-24T09:25:10.077Z
6454e5ac273f64983024ba5d
/avatars/ca6f861ed830a79f7a1eba04ebe84afc.svg
Vatolin Alexey
user
vatolinalex
67b6fa9cb544aa153178a63b
Nandan Thakur
claimed_verified
2025-02-24T09:24:59.423Z
60196690dd31fde3c1062960
https://cdn-avatars.huggingface.co/v1/production/uploads/1612277330660-noauth.jpeg
Nandan Thakur
user
nthakur
67b6fa9cb544aa153178a63c
Manan Dey
67b6fa9cb544aa153178a63d
Dipam Vasani
67b6fa9cb544aa153178a63e
Pranjal Chitale
67b6fa9cb544aa153178a63f
Simone Tedeschi
claimed_verified
2025-02-24T09:23:51.243Z
61b85aa99ba538c73a7dc78b
https://cdn-avatars.huggingface.co/v1/production/uploads/61b85aa99ba538c73a7dc78b/gWxtQAvOYn7cXgE_nAy0p.jpeg
Simone Tedeschi
user
sted97
67b6fa9cb544aa153178a640
Nguyen Tai
67b6fa9cb544aa153178a641
Artem Snegirev
67b6fa9cb544aa153178a642
Michael Günther
67b6fa9cb544aa153178a643
Mengzhou Xia
67b6fa9cb544aa153178a644
Weijia Shi
67b6fa9cb544aa153178a645
Xing Han Lù
claimed_verified
2025-02-21T16:15:18.716Z
5fa9ff3ea13e063b8b2b60cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1633380224986-5fa9ff3ea13e063b8b2b60cb.jpeg
Xing Han Lù
user
xhluca
67b6fa9cb544aa153178a646
Jordan Clive
claimed_verified
2025-05-16T07:13:15.019Z
608995d619137b3a6ba76113
https://cdn-avatars.huggingface.co/v1/production/uploads/608995d619137b3a6ba76113/QK25kNi5XIlw8m3104KC8.jpeg
Jordan Clive
user
jordiclive
67b6fa9cb544aa153178a647
Gayatri Krishnakumar
67b6fa9cb544aa153178a648
Anna Maksimova
67b6fa9cb544aa153178a649
Silvan Wehrli
67b6fa9cb544aa153178a64a
Maria Tikhonova
67b6fa9cb544aa153178a64b
Henil Panchal
67b6fa9cb544aa153178a64c
Aleksandr Abramov
67b6fa9cb544aa153178a64d
Malte Ostendorff
claimed_verified
2025-02-21T16:15:16.649Z
5efda656ff69163f6f59e5d2
https://cdn-avatars.huggingface.co/v1/production/uploads/5efda656ff69163f6f59e5d2/ru2nfhaNjB9-Ls_vbMq92.jpeg
malteos
user
malteos
67b6fa9cb544aa153178a64e
Zheng Liu
67b6fa9cb544aa153178a64f
Simon Clematide
67b6fa9cb544aa153178a650
Lester James Miranda
claimed_verified
2025-03-12T20:46:15.143Z
634e20a0c1ce28f1de920cc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
Lj V. Miranda
user
ljvmiranda921
67b6fa9cb544aa153178a651
Alena Fenogenova
67b6fa9cb544aa153178a652
Guangyu Song
67b6fa9cb544aa153178a653
Ruqiya Bin Safi
67b6fa9cb544aa153178a654
Wen-Ding Li
67b6fa9cb544aa153178a655
Alessia Borghini
67b6fa9cb544aa153178a656
Federico Cassano
67b6fa9cb544aa153178a657
Hongjin Su
67b6fa9cb544aa153178a658
Jimmy Lin
67b6fa9cb544aa153178a659
Howard Yen
67b6fa9cb544aa153178a65a
Lasse Hansen
67b6fa9cb544aa153178a65b
Sara Hooker
67b6fa9cb544aa153178a65c
Chenghao Xiao
claimed_verified
2025-04-16T09:24:54.185Z
63108cc834c7d77420b0fd68
/avatars/2721e573a417a8ec0b81ee048c4b42ba.svg
chenghao xiao
user
gowitheflow
67b6fa9cb544aa153178a65d
Vaibhav Adlakha
claimed_verified
2025-02-21T16:15:20.849Z
61981af5d420757268e195ac
/avatars/8b59aaf33447224f83d497425fd7ea8f.svg
Vaibhav Adlakha
user
vaibhavad
67b6fa9cb544aa153178a65e
Orion Weller
67b6fa9cb544aa153178a65f
Siva Reddy
67b6fa9cb544aa153178a660
Niklas Muennighoff
MMTEB: Massive Multilingual Text Embedding Benchmark
Text embeddings are typically evaluated on a limited set of tasks, which are
constrained by language, domain, and task diversity. To address these
limitations and provide a more comprehensive evaluation, we introduce the
Massive Multilingual Text Embedding Benchmark (MMTEB) - a large-scale,
community-driven expansion of MTEB, covering over 500 quality-controlled
evaluation tasks across 250+ languages. MMTEB includes a diverse set of
challenging, novel tasks such as instruction following, long-document
retrieval, and code retrieval, representing the largest multilingual collection
of evaluation tasks for embedding models to date. Using this collection, we
develop several highly multilingual benchmarks, which we use to evaluate a
representative set of models. We find that while large language models (LLMs)
with billions of parameters can achieve state-of-the-art performance on certain
language subsets and task categories, the best-performing publicly available
model is multilingual-e5-large-instruct with only 560 million parameters. To
facilitate accessibility and reduce computational cost, we introduce a novel
downsampling method based on inter-task correlation, ensuring a diverse
selection while preserving relative model rankings. Furthermore, we optimize
tasks such as retrieval by sampling hard negatives, creating smaller but
effective splits. These optimizations allow us to introduce benchmarks that
drastically reduce computational demands. For instance, our newly introduced
zero-shot English benchmark maintains a ranking order similar to the full-scale
version but at a fraction of the computational cost.
The MMTEB benchmark expands MTEB with over 500 tasks across 250+ languages to comprehensively evaluate text embeddings, finding that smaller multilingual models can outperform large LLMs in many cases.
text embeddings Massive Multilingual Text Embedding Benchmark (MMTEB) MTEB multilingual instruction following long-document retrieval code retrieval large language models (LLMs) multilingual-e5-large-instruct downsampling method inter-task correlation hard negatives zero-shot benchmark