GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge
  Benchmarking
Large Vision-Language Models (LVLMs) have recently gained attention due to
their distinctive performance and broad applicability. While it has been
previously shown that their efficacy in usage scenarios involving non-Western
contexts falls short, existing studies are limited in scope, covering just a
narrow range of cultures, focusing exclusively on a small number of cultural
aspects, or evaluating a limited selection of models on a single task only.
Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive
multimodal benchmark designed to assess a broad spectrum of cultural knowledge
across 144 countries representing six global macro-regions. GIMMICK comprises
six tasks built upon three new datasets that span 728 unique cultural events or
facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary
and 26 open-weight models of all sizes. We systematically examine (1) regional
cultural biases, (2) the influence of model size, (3) input modalities, and (4)
external cues. Our analyses reveal strong biases toward Western cultures across
models and tasks and highlight strong correlations between model size and
performance, as well as the effectiveness of multimodal input and external
geographic cues. We further find that models have more knowledge of tangible
than intangible aspects (e.g., food vs. rituals) and that they excel in
recognizing broad cultural origins but struggle with a more nuanced
understanding.
67b6faf5a96bf2b8ff8c235c
Florian Schneider
claimed_verified
2025-02-20T10:49:37.443Z
62dfd54798815401141c47fe
https://cdn-avatars.huggingface.co/v1/production/uploads/62dfd54798815401141c47fe/ct2OA_K0Wwpshy8DCswxy.png
Flo Schneider
user
floschne
67b6faf5a96bf2b8ff8c235d
Carolin Holtermann
67b6faf5a96bf2b8ff8c235e
Chris Biemann
67b6faf5a96bf2b8ff8c235f
Anne Lauscher
admin_assigned
2025-02-20T17:41:05.177Z
626c02e7703f3b27dd590896
https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
Anne Lauscher
user
anlausch
GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge
  Benchmarking
Large Vision-Language Models (LVLMs) have recently gained attention due to
their distinctive performance and broad applicability. While it has been
previously shown that their efficacy in usage scenarios involving non-Western
contexts falls short, existing studies are limited in scope, covering just a
narrow range of cultures, focusing exclusively on a small number of cultural
aspects, or evaluating a limited selection of models on a single task only.
Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive
multimodal benchmark designed to assess a broad spectrum of cultural knowledge
across 144 countries representing six global macro-regions. GIMMICK comprises
six tasks built upon three new datasets that span 728 unique cultural events or
facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary
and 26 open-weight models of all sizes. We systematically examine (1) regional
cultural biases, (2) the influence of model size, (3) input modalities, and (4)
external cues. Our analyses reveal strong biases toward Western cultures across
models and tasks and highlight strong correlations between model size and
performance, as well as the effectiveness of multimodal input and external
geographic cues. We further find that models have more knowledge of tangible
than intangible aspects (e.g., food vs. rituals) and that they excel in
recognizing broad cultural origins but struggle with a more nuanced
understanding.
GIMMICK, a comprehensive multimodal benchmark, assesses the cultural knowledge of large vision-language models across 144 countries, revealing significant biases towards Western cultures and differences in performance based on model size and input type.
Large Vision-Language Models (LVLMs) GIMMICK multimodal benchmark cultural knowledge global macro-regions datasets cultural events cultural facets input modalities external cues regional cultural biases model size tangible aspects intangible aspects cultural origins