title: GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge
    Benchmarking
summary: Large Vision-Language Models (LVLMs) have recently gained attention due to
  their distinctive performance and broad applicability. While it has been
  previously shown that their efficacy in usage scenarios involving non-Western
  contexts falls short, existing studies are limited in scope, covering just a
  narrow range of cultures, focusing exclusively on a small number of cultural
  aspects, or evaluating a limited selection of models on a single task only.
  Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive
  multimodal benchmark designed to assess a broad spectrum of cultural knowledge
  across 144 countries representing six global macro-regions. GIMMICK comprises
  six tasks built upon three new datasets that span 728 unique cultural events or
  facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary
  and 26 open-weight models of all sizes. We systematically examine (1) regional
  cultural biases, (2) the influence of model size, (3) input modalities, and (4)
  external cues. Our analyses reveal strong biases toward Western cultures across
  models and tasks and highlight strong correlations between model size and
  performance, as well as the effectiveness of multimodal input and external
  geographic cues. We further find that models have more knowledge of tangible
  than intangible aspects (e.g., food vs. rituals) and that they excel in
  recognizing broad cultural origins but struggle with a more nuanced
  understanding.
paper_authors._id: 67b6faf5a96bf2b8ff8c235c
paper_authors.name: Florian Schneider
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-20T10:49:37.443Z
paper_authors._id: 62dfd54798815401141c47fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62dfd54798815401141c47fe/ct2OA_K0Wwpshy8DCswxy.png
paper_authors.fullname: Flo Schneider
paper_authors.type: user
paper_authors.user: floschne
paper_authors._id: 67b6faf5a96bf2b8ff8c235d
paper_authors.name: Carolin Holtermann
paper_authors._id: 67b6faf5a96bf2b8ff8c235e
paper_authors.name: Chris Biemann
paper_authors._id: 67b6faf5a96bf2b8ff8c235f
paper_authors.name: Anne Lauscher
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-20T17:41:05.177Z
paper_authors._id: 626c02e7703f3b27dd590896
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654503075060-626c02e7703f3b27dd590896.jpeg
paper_authors.fullname: Anne Lauscher
paper_authors.type: user
paper_authors.user: anlausch
paper_title: GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge
    Benchmarking
paper_summary: Large Vision-Language Models (LVLMs) have recently gained attention due to
  their distinctive performance and broad applicability. While it has been
  previously shown that their efficacy in usage scenarios involving non-Western
  contexts falls short, existing studies are limited in scope, covering just a
  narrow range of cultures, focusing exclusively on a small number of cultural
  aspects, or evaluating a limited selection of models on a single task only.
  Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive
  multimodal benchmark designed to assess a broad spectrum of cultural knowledge
  across 144 countries representing six global macro-regions. GIMMICK comprises
  six tasks built upon three new datasets that span 728 unique cultural events or
  facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary
  and 26 open-weight models of all sizes. We systematically examine (1) regional
  cultural biases, (2) the influence of model size, (3) input modalities, and (4)
  external cues. Our analyses reveal strong biases toward Western cultures across
  models and tasks and highlight strong correlations between model size and
  performance, as well as the effectiveness of multimodal input and external
  geographic cues. We further find that models have more knowledge of tangible
  than intangible aspects (e.g., food vs. rituals) and that they excel in
  recognizing broad cultural origins but struggle with a more nuanced
  understanding.
paper_ai_summary: GIMMICK, a comprehensive multimodal benchmark, assesses the cultural knowledge of large vision-language models across 144 countries, revealing significant biases towards Western cultures and differences in performance based on model size and input type.
paper_ai_keywords: Large Vision-Language Models (LVLMs) GIMMICK multimodal benchmark cultural knowledge global macro-regions datasets cultural events cultural facets input modalities external cues regional cultural biases model size tangible aspects intangible aspects cultural origins