title: All Languages Matter: Evaluating LMMs on Culturally Diverse 100
    Languages
summary: Existing Large Multimodal Models (LMMs) generally focus on only a few regions
  and languages. As LMMs continue to improve, it is increasingly important to
  ensure they understand cultural contexts, respect local sensitivities, and
  support low-resource languages, all while effectively integrating corresponding
  visual cues. In pursuit of culturally diverse global multimodal models, our
  proposed All Languages Matter Benchmark (ALM-bench) represents the largest and
  most comprehensive effort to date for evaluating LMMs across 100 languages.
  ALM-bench challenges existing models by testing their ability to understand and
  reason about culturally diverse images paired with text in various languages,
  including many low-resource languages traditionally underrepresented in LMM
  research. The benchmark offers a robust and nuanced evaluation framework
  featuring various question formats, including true/false, multiple choice, and
  open-ended questions, which are further divided into short and long-answer
  categories. ALM-bench design ensures a comprehensive assessment of a model's
  ability to handle varied levels of difficulty in visual and linguistic
  reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully
  curates content from 13 distinct cultural aspects, ranging from traditions and
  rituals to famous personalities and celebrations. Through this, ALM-bench not
  only provides a rigorous testing ground for state-of-the-art open and
  closed-source LMMs but also highlights the importance of cultural and
  linguistic inclusivity, encouraging the development of models that can serve
  diverse global populations effectively. Our benchmark is publicly available.
paper_authors._id: 6745a57662eb3714377eef30
paper_authors.name: Ashmal Vayani
paper_authors._id: 6745a57662eb3714377eef31
paper_authors.name: Dinura Dissanayake
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:40:00.685Z
paper_authors._id: 6639cd849662bb58d5fe5793
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6639cd849662bb58d5fe5793/PHUa38L5_llipN52ngajY.png
paper_authors.fullname: Dissanayake
paper_authors.type: user
paper_authors.user: Dinura
paper_authors._id: 6745a57662eb3714377eef32
paper_authors.name: Hasindri Watawana
paper_authors._id: 6745a57662eb3714377eef33
paper_authors.name: Noor Ahsan
paper_authors._id: 6745a57662eb3714377eef34
paper_authors.name: Nevasini Sasikumar
paper_authors._id: 6745a57662eb3714377eef35
paper_authors.name: Omkar Thawakar
paper_authors._id: 6745a57662eb3714377eef36
paper_authors.name: Henok Biadglign Ademtew
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-12T09:19:13.428Z
paper_authors._id: 62c488530a06c03931610fe0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c488530a06c03931610fe0/Zi5mTvE-R0bHOjuUEl56k.jpeg
paper_authors.fullname: Henok Ademtew
paper_authors.type: user
paper_authors.user: Henok
paper_authors._id: 6745a57662eb3714377eef37
paper_authors.name: Yahya Hmaiti
paper_authors._id: 6745a57662eb3714377eef38
paper_authors.name: Amandeep Kumar
paper_authors._id: 6745a57662eb3714377eef39
paper_authors.name: Kartik Kuckreja
paper_authors._id: 6745a57662eb3714377eef3a
paper_authors.name: Mykola Maslych
paper_authors._id: 6745a57662eb3714377eef3b
paper_authors.name: Wafa Al Ghallabi
paper_authors._id: 6745a57662eb3714377eef3c
paper_authors.name: Mihail Mihaylov
paper_authors._id: 6745a57662eb3714377eef3d
paper_authors.name: Chao Qin
paper_authors._id: 6745a57662eb3714377eef3e
paper_authors.name: Abdelrahman M Shaker
paper_authors._id: 6745a57662eb3714377eef3f
paper_authors.name: Mike Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T21:15:26.163Z
paper_authors._id: 60d33fbbd7b174177faabd4f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60d33fbbd7b174177faabd4f/pfyv_xj2B2m2N4F4sT9zJ.jpeg
paper_authors.fullname: Mike Zhang
paper_authors.type: user
paper_authors.user: jjzha
paper_authors._id: 6745a57662eb3714377eef40
paper_authors.name: Mahardika Krisna Ihsani
paper_authors._id: 6745a57662eb3714377eef41
paper_authors.name: Amiel Esplana
paper_authors._id: 6745a57662eb3714377eef42
paper_authors.name: Monil Gokani
paper_authors._id: 6745a57662eb3714377eef43
paper_authors.name: Shachar Mirkin
paper_authors._id: 6745a57662eb3714377eef44
paper_authors.name: Harsh Singh
paper_authors._id: 6745a57662eb3714377eef45
paper_authors.name: Ashay Srivastava
paper_authors._id: 6745a57662eb3714377eef46
paper_authors.name: Endre Hamerlik
paper_authors._id: 6745a57662eb3714377eef47
paper_authors.name: Fathinah Asma Izzati
paper_authors._id: 6745a57662eb3714377eef48
paper_authors.name: Fadillah Adamsyah Maani
paper_authors._id: 6745a57662eb3714377eef49
paper_authors.name: Sebastian Cavada
paper_authors._id: 6745a57662eb3714377eef4a
paper_authors.name: Jenny Chim
paper_authors._id: 6745a57662eb3714377eef4b
paper_authors.name: Rohit Gupta
paper_authors._id: 6745a57662eb3714377eef4c
paper_authors.name: Sanjay Manjunath
paper_authors._id: 6745a57662eb3714377eef4d
paper_authors.name: Kamila Zhumakhanova
paper_authors._id: 6745a57662eb3714377eef4e
paper_authors.name: Feno Heriniaina Rabevohitra
paper_authors._id: 6745a57662eb3714377eef4f
paper_authors.name: Azril Amirudin
paper_authors._id: 6745a57662eb3714377eef50
paper_authors.name: Muhammad Ridzuan
paper_authors._id: 6745a57662eb3714377eef51
paper_authors.name: Daniya Kareem
paper_authors._id: 6745a57662eb3714377eef52
paper_authors.name: Ketan More
paper_authors._id: 6745a57662eb3714377eef53
paper_authors.name: Kunyang Li
paper_authors._id: 6745a57662eb3714377eef54
paper_authors.name: Pramesh Shakya
paper_authors._id: 6745a57662eb3714377eef55
paper_authors.name: Muhammad Saad
paper_authors._id: 6745a57662eb3714377eef56
paper_authors.name: Amirpouya Ghasemaghaei
paper_authors._id: 6745a57662eb3714377eef57
paper_authors.name: Amirbek Djanibekov
paper_authors._id: 6745a57662eb3714377eef58
paper_authors.name: Dilshod Azizov
paper_authors._id: 6745a57662eb3714377eef59
paper_authors.name: Branislava Jankovic
paper_authors._id: 6745a57662eb3714377eef5a
paper_authors.name: Naman Bhatia
paper_authors._id: 6745a57662eb3714377eef5b
paper_authors.name: Alvaro Cabrera
paper_authors._id: 6745a57662eb3714377eef5c
paper_authors.name: Johan Obando-Ceron
paper_authors._id: 6745a57662eb3714377eef5d
paper_authors.name: Olympiah Otieno
paper_authors._id: 6745a57662eb3714377eef5e
paper_authors.name: Fabian Farestam
paper_authors._id: 6745a57662eb3714377eef5f
paper_authors.name: Muztoba Rabbani
paper_authors._id: 6745a57662eb3714377eef60
paper_authors.name: Sanoojan Baliah
paper_authors._id: 6745a57662eb3714377eef61
paper_authors.name: Santosh Sanjeev
paper_authors._id: 6745a57662eb3714377eef62
paper_authors.name: Abduragim Shtanchaev
paper_authors._id: 6745a57662eb3714377eef63
paper_authors.name: Maheen Fatima
paper_authors._id: 6745a57662eb3714377eef64
paper_authors.name: Thao Nguyen
paper_authors._id: 6745a57662eb3714377eef65
paper_authors.name: Amrin Kareem
paper_authors._id: 6745a57662eb3714377eef66
paper_authors.name: Toluwani Aremu
paper_authors._id: 6745a57662eb3714377eef67
paper_authors.name: Nathan Xavier
paper_authors._id: 6745a57662eb3714377eef68
paper_authors.name: Amit Bhatkal
paper_authors._id: 6745a57662eb3714377eef69
paper_authors.name: Hawau Toyin
paper_authors._id: 6745a57662eb3714377eef6a
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:43.573Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 6745a57662eb3714377eef6b
paper_authors.name: Hisham Cholakkal
paper_authors._id: 6745a57662eb3714377eef6c
paper_authors.name: Rao Muhammad Anwer
paper_authors._id: 6745a57662eb3714377eef6d
paper_authors.name: Michael Felsberg
paper_authors._id: 6745a57662eb3714377eef6e
paper_authors.name: Jorma Laaksonen
paper_authors._id: 6745a57662eb3714377eef6f
paper_authors.name: Thamar Solorio
paper_authors._id: 6745a57662eb3714377eef70
paper_authors.name: Monojit Choudhury
paper_authors._id: 6745a57662eb3714377eef71
paper_authors.name: Ivan Laptev
paper_authors._id: 6745a57662eb3714377eef72
paper_authors.name: Mubarak Shah
paper_authors._id: 6745a57662eb3714377eef73
paper_authors.name: Salman Khan
paper_authors._id: 6745a57662eb3714377eef74
paper_authors.name: Fahad Khan
paper_title: All Languages Matter: Evaluating LMMs on Culturally Diverse 100
    Languages
paper_summary: Existing Large Multimodal Models (LMMs) generally focus on only a few regions
  and languages. As LMMs continue to improve, it is increasingly important to
  ensure they understand cultural contexts, respect local sensitivities, and
  support low-resource languages, all while effectively integrating corresponding
  visual cues. In pursuit of culturally diverse global multimodal models, our
  proposed All Languages Matter Benchmark (ALM-bench) represents the largest and
  most comprehensive effort to date for evaluating LMMs across 100 languages.
  ALM-bench challenges existing models by testing their ability to understand and
  reason about culturally diverse images paired with text in various languages,
  including many low-resource languages traditionally underrepresented in LMM
  research. The benchmark offers a robust and nuanced evaluation framework
  featuring various question formats, including true/false, multiple choice, and
  open-ended questions, which are further divided into short and long-answer
  categories. ALM-bench design ensures a comprehensive assessment of a model's
  ability to handle varied levels of difficulty in visual and linguistic
  reasoning. To capture the rich tapestry of global cultures, ALM-bench carefully
  curates content from 13 distinct cultural aspects, ranging from traditions and
  rituals to famous personalities and celebrations. Through this, ALM-bench not
  only provides a rigorous testing ground for state-of-the-art open and
  closed-source LMMs but also highlights the importance of cultural and
  linguistic inclusivity, encouraging the development of models that can serve
  diverse global populations effectively. Our benchmark is publicly available.
paper_ai_keywords: Large Multimodal Models (LMMs) All Languages Matter Benchmark (ALM-bench) culturally diverse global multimodal models visually diverse images true/false questions multiple choice questions open-ended questions multicultural content diverse global populations