title: MVL-SIB: A Massively Multilingual Vision-Language Benchmark for
    Cross-Modal Topical Matching
summary: Existing multilingual vision-language (VL) benchmarks often only cover a
  handful of languages. Consequently, evaluations of large vision-language models
  (LVLMs) predominantly target high-resource languages, underscoring the need for
  evaluation data for low-resource languages. To address this limitation, we
  introduce MVL-SIB, a massively multilingual vision-language benchmark that
  evaluates both cross-modal and text-only topical matching across 205 languages
  -- over 100 more than the most multilingual existing VL benchmarks encompass.
  We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini)
  on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic
  matching in lower-resource languages, performing no better than chance on
  languages like N'Koo. Our analysis further reveals that VL support in LVLMs
  declines disproportionately relative to textual support for lower-resource
  languages, as evidenced by comparison of cross-modal and text-only topical
  matching performance. We further observe that open-weight LVLMs do not benefit
  from representing a topic with more than one image, suggesting that these
  models are not yet fully effective at handling multi-image tasks. By
  correlating performance on MVL-SIB with other multilingual VL benchmarks, we
  highlight that MVL-SIB serves as a comprehensive probe of multilingual VL
  understanding in LVLMs.
paper_authors._id: 67b5b31f5a17526b55c3ccde
paper_authors.name: Fabian David Schmidt
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T15:53:15.852Z
paper_authors._id: 64c8c2d87d0ea4e7f12995c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c8c2d87d0ea4e7f12995c6/h8eWJrz8kqavemy8vQ2NK.jpeg
paper_authors.fullname: Fabian David Schmidt
paper_authors.type: user
paper_authors.user: fdschmidt93
paper_authors._id: 67b5b31f5a17526b55c3ccdf
paper_authors.name: Florian Schneider
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T10:49:49.711Z
paper_authors._id: 62dfd54798815401141c47fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62dfd54798815401141c47fe/ct2OA_K0Wwpshy8DCswxy.png
paper_authors.fullname: Flo Schneider
paper_authors.type: user
paper_authors.user: floschne
paper_authors._id: 67b5b31f5a17526b55c3cce0
paper_authors.name: Chris Biemann
paper_authors._id: 67b5b31f5a17526b55c3cce1
paper_authors.name: Goran Glavaš
paper_title: MVL-SIB: A Massively Multilingual Vision-Language Benchmark for
    Cross-Modal Topical Matching
paper_summary: Existing multilingual vision-language (VL) benchmarks often only cover a
  handful of languages. Consequently, evaluations of large vision-language models
  (LVLMs) predominantly target high-resource languages, underscoring the need for
  evaluation data for low-resource languages. To address this limitation, we
  introduce MVL-SIB, a massively multilingual vision-language benchmark that
  evaluates both cross-modal and text-only topical matching across 205 languages
  -- over 100 more than the most multilingual existing VL benchmarks encompass.
  We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini)
  on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic
  matching in lower-resource languages, performing no better than chance on
  languages like N'Koo. Our analysis further reveals that VL support in LVLMs
  declines disproportionately relative to textual support for lower-resource
  languages, as evidenced by comparison of cross-modal and text-only topical
  matching performance. We further observe that open-weight LVLMs do not benefit
  from representing a topic with more than one image, suggesting that these
  models are not yet fully effective at handling multi-image tasks. By
  correlating performance on MVL-SIB with other multilingual VL benchmarks, we
  highlight that MVL-SIB serves as a comprehensive probe of multilingual VL
  understanding in LVLMs.
paper_ai_summary: MVL-SIB, a multilingual vision-language benchmark spanning 205 languages, reveals that existing LVLMs underperform in cross-modal topic matching for low-resource languages.
paper_ai_keywords: multilingual vision-language benchmarks LVLMs cross-modal matching text-only matching GPT-4o(-mini) lower-resource languages N'Koo VL support textual support multi-image tasks