All Languages Matter: Evaluating LMMs on Culturally Diverse 100
  Languages
Existing Large Multimodal Models (LMMs) generally focus on only a few regions
and languages. As LMMs continue to improve, it is increasingly important to
ensure they understand cultural contexts, respect local sensitivities, and
support low-resource languages, all while effectively integrating corresponding
visual cues. In pursuit of culturally diverse global multimodal models, our
proposed All Languages Matter Benchmark (ALM-bench) represents the largest and
most comprehensive effort to date for evaluating LMMs across 100 languages.
ALM-bench challenges existing models by testing their ability to understand and
reason about culturally diverse images paired with text in various languages,
including many low-resource languages traditionally underrepresented in LMM
research. The benchmark offers a robust and nuanced evaluation framework
featuring various question formats, including true/false, multiple choice, and
open-ended questions, which are further divided into short and long-answer
categories. ALM-bench design ensures a comprehensive assessment of a model's
ability to handle varied levels of difficulty in visual and linguistic
reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully
curates content from 13 distinct cultural aspects, ranging from traditions and
rituals to famous personalities and celebrations. Through this, ALM-bench not
only provides a rigorous testing ground for state-of-the-art open and
closed-source LMMs but also highlights the importance of cultural and
linguistic inclusivity, encouraging the development of models that can serve
diverse global populations effectively. Our benchmark is publicly available.
6745a57662eb3714377eef30
Ashmal Vayani
6745a57662eb3714377eef31
Dinura Dissanayake
claimed_verified
2025-02-04T09:40:00.685Z
6639cd849662bb58d5fe5793
https://cdn-avatars.huggingface.co/v1/production/uploads/6639cd849662bb58d5fe5793/PHUa38L5_llipN52ngajY.png
Dissanayake
user
Dinura
6745a57662eb3714377eef32
Hasindri Watawana
6745a57662eb3714377eef33
Noor Ahsan
6745a57662eb3714377eef34
Nevasini Sasikumar
6745a57662eb3714377eef35
Omkar Thawakar
6745a57662eb3714377eef36
Henok Biadglign Ademtew
claimed_verified
2025-02-12T09:19:13.428Z
62c488530a06c03931610fe0
https://cdn-avatars.huggingface.co/v1/production/uploads/62c488530a06c03931610fe0/Zi5mTvE-R0bHOjuUEl56k.jpeg
Henok Ademtew
user
Henok
6745a57662eb3714377eef37
Yahya Hmaiti
6745a57662eb3714377eef38
Amandeep Kumar
6745a57662eb3714377eef39
Kartik Kuckreja
claimed_verified
2025-06-24T08:15:16.392Z
64f3114bd818f024d9950642
/avatars/9d0129d85264592251a2e8fab05e02cf.svg
kuckreja
user
kartik060702
6745a57662eb3714377eef3a
Mykola Maslych
6745a57662eb3714377eef3b
Wafa Al Ghallabi
6745a57662eb3714377eef3c
Mihail Mihaylov
6745a57662eb3714377eef3d
Chao Qin
6745a57662eb3714377eef3e
Abdelrahman M Shaker
6745a57662eb3714377eef3f
Mike Zhang
claimed_verified
2024-11-28T21:15:26.163Z
60d33fbbd7b174177faabd4f
https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg
Mike Zhang
user
jjzha
6745a57662eb3714377eef40
Mahardika Krisna Ihsani
6745a57662eb3714377eef41
Amiel Esplana
6745a57662eb3714377eef42
Monil Gokani
6745a57662eb3714377eef43
Shachar Mirkin
6745a57662eb3714377eef44
Harsh Singh
6745a57662eb3714377eef45
Ashay Srivastava
6745a57662eb3714377eef46
Endre Hamerlik
6745a57662eb3714377eef47
Fathinah Asma Izzati
6745a57662eb3714377eef48
Fadillah Adamsyah Maani
6745a57662eb3714377eef49
Sebastian Cavada
6745a57662eb3714377eef4a
Jenny Chim
6745a57662eb3714377eef4b
Rohit Gupta
6745a57662eb3714377eef4c
Sanjay Manjunath
6745a57662eb3714377eef4d
Kamila Zhumakhanova
6745a57662eb3714377eef4e
Feno Heriniaina Rabevohitra
6745a57662eb3714377eef4f
Azril Amirudin
6745a57662eb3714377eef50
Muhammad Ridzuan
6745a57662eb3714377eef51
Daniya Kareem
6745a57662eb3714377eef52
Ketan More
6745a57662eb3714377eef53
Kunyang Li
6745a57662eb3714377eef54
Pramesh Shakya
6745a57662eb3714377eef55
Muhammad Saad
6745a57662eb3714377eef56
Amirpouya Ghasemaghaei
6745a57662eb3714377eef57
Amirbek Djanibekov
6745a57662eb3714377eef58
Dilshod Azizov
6745a57662eb3714377eef59
Branislava Jankovic
6745a57662eb3714377eef5a
Naman Bhatia
6745a57662eb3714377eef5b
Alvaro Cabrera
6745a57662eb3714377eef5c
Johan Obando-Ceron
6745a57662eb3714377eef5d
Olympiah Otieno
6745a57662eb3714377eef5e
Fabian Farestam
claimed_verified
2025-04-01T15:59:53.780Z
649edc8ca74a8d2dc44f117e
/avatars/33064f5dd5c8029278cdaa01cd16d4ab.svg
Fabian Farestam
user
northern-64bit
6745a57662eb3714377eef5f
Muztoba Rabbani
6745a57662eb3714377eef60
Sanoojan Baliah
6745a57662eb3714377eef61
Santosh Sanjeev
6745a57662eb3714377eef62
Abduragim Shtanchaev
6745a57662eb3714377eef63
Maheen Fatima
6745a57662eb3714377eef64
Thao Nguyen
6745a57662eb3714377eef65
Amrin Kareem
6745a57662eb3714377eef66
Toluwani Aremu
6745a57662eb3714377eef67
Nathan Xavier
6745a57662eb3714377eef68
Amit Bhatkal
6745a57662eb3714377eef69
Hawau Toyin
claimed_verified
2025-06-10T09:31:40.562Z
64457216b272430bdbf3f7df
https://cdn-avatars.huggingface.co/v1/production/uploads/64457216b272430bdbf3f7df/Dpz8pnPMMn83PxjNR5Bpf.jpeg
Hawau Olamide Toyin
user
herwoww
6745a57662eb3714377eef6a
Aman Chadha
claimed_verified
2024-11-27T22:20:43.573Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
6745a57662eb3714377eef6b
Hisham Cholakkal
6745a57662eb3714377eef6c
Rao Muhammad Anwer
6745a57662eb3714377eef6d
Michael Felsberg
6745a57662eb3714377eef6e
Jorma Laaksonen
6745a57662eb3714377eef6f
Thamar Solorio
6745a57662eb3714377eef70
Monojit Choudhury
6745a57662eb3714377eef71
Ivan Laptev
6745a57662eb3714377eef72
Mubarak Shah
6745a57662eb3714377eef73
Salman Khan
6745a57662eb3714377eef74
Fahad Khan
All Languages Matter: Evaluating LMMs on Culturally Diverse 100
  Languages
Existing Large Multimodal Models (LMMs) generally focus on only a few regions
and languages. As LMMs continue to improve, it is increasingly important to
ensure they understand cultural contexts, respect local sensitivities, and
support low-resource languages, all while effectively integrating corresponding
visual cues. In pursuit of culturally diverse global multimodal models, our
proposed All Languages Matter Benchmark (ALM-bench) represents the largest and
most comprehensive effort to date for evaluating LMMs across 100 languages.
ALM-bench challenges existing models by testing their ability to understand and
reason about culturally diverse images paired with text in various languages,
including many low-resource languages traditionally underrepresented in LMM
research. The benchmark offers a robust and nuanced evaluation framework
featuring various question formats, including true/false, multiple choice, and
open-ended questions, which are further divided into short and long-answer
categories. ALM-bench design ensures a comprehensive assessment of a model's
ability to handle varied levels of difficulty in visual and linguistic
reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully
curates content from 13 distinct cultural aspects, ranging from traditions and
rituals to famous personalities and celebrations. Through this, ALM-bench not
only provides a rigorous testing ground for state-of-the-art open and
closed-source LMMs but also highlights the importance of cultural and
linguistic inclusivity, encouraging the development of models that can serve
diverse global populations effectively. Our benchmark is publicly available.
ALM-bench provides a comprehensive evaluation of Large Multimodal Models across 100 languages, focusing on cultural and linguistic inclusivity.
Large Multimodal Models LMMs ALM-bench culturally diverse images low-resource languages visual and linguistic reasoning true/false questions multiple choice open-ended questions short and long-answer categories cultural aspects traditions rituals famous personalities celebrations global cultures cultural and linguistic inclusivity