title: RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture
    Understanding
summary: As vision-language models (VLMs) become increasingly integrated into daily
  life, the need for accurate visual culture understanding is becoming critical.
  Yet, these models frequently fall short in interpreting cultural nuances
  effectively. Prior work has demonstrated the effectiveness of
  retrieval-augmented generation (RAG) in enhancing cultural understanding in
  text-only settings, while its application in multimodal scenarios remains
  underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented
  Visual culturE uNdErstAnding), a new benchmark designed to advance visual
  culture understanding through retrieval, focusing on two tasks: culture-focused
  visual question answering (cVQA) and culture-informed image captioning (cIC).
  RAVENEA extends existing datasets by integrating over 10,000 Wikipedia
  documents curated and ranked by human annotators. With RAVENEA, we train and
  evaluate seven multimodal retrievers for each image query, and measure the
  downstream impact of retrieval-augmented inputs across fourteen
  state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented
  with culture-aware retrieval, outperform their non-augmented counterparts (by
  at least 3.2% absolute on cVQA and 6.2% absolute on cIC). This highlights the
  value of retrieval-augmented methods and culturally inclusive benchmarks for
  multimodal understanding.
paper_authors._id: 682f38862783867b8e775d27
paper_authors.name: Jiaang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:17:28.360Z
paper_authors._id: 6686658221f096c7fb4e0b24
paper_authors.avatarUrl: /avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
paper_authors.fullname: Jiaang Li
paper_authors.type: user
paper_authors.user: jaagli
paper_authors._id: 682f38862783867b8e775d28
paper_authors.name: Yifei Yuan
paper_authors._id: 682f38862783867b8e775d29
paper_authors.name: Wenyan Li
paper_authors._id: 682f38862783867b8e775d2a
paper_authors.name: Mohammad Aliannejadi
paper_authors._id: 682f38862783867b8e775d2b
paper_authors.name: Daniel Hershcovich
paper_authors._id: 682f38862783867b8e775d2c
paper_authors.name: Anders Søgaard
paper_authors._id: 682f38862783867b8e775d2d
paper_authors.name: Ivan Vulić
paper_authors._id: 682f38862783867b8e775d2e
paper_authors.name: Wenxuan Zhang
paper_authors._id: 682f38862783867b8e775d2f
paper_authors.name: Paul Pu Liang
paper_authors._id: 682f38862783867b8e775d30
paper_authors.name: Yang Deng
paper_authors._id: 682f38862783867b8e775d31
paper_authors.name: Serge Belongie
paper_title: RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture
    Understanding
paper_summary: As vision-language models (VLMs) become increasingly integrated into daily
  life, the need for accurate visual culture understanding is becoming critical.
  Yet, these models frequently fall short in interpreting cultural nuances
  effectively. Prior work has demonstrated the effectiveness of
  retrieval-augmented generation (RAG) in enhancing cultural understanding in
  text-only settings, while its application in multimodal scenarios remains
  underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented
  Visual culturE uNdErstAnding), a new benchmark designed to advance visual
  culture understanding through retrieval, focusing on two tasks: culture-focused
  visual question answering (cVQA) and culture-informed image captioning (cIC).
  RAVENEA extends existing datasets by integrating over 10,000 Wikipedia
  documents curated and ranked by human annotators. With RAVENEA, we train and
  evaluate seven multimodal retrievers for each image query, and measure the
  downstream impact of retrieval-augmented inputs across fourteen
  state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented
  with culture-aware retrieval, outperform their non-augmented counterparts (by
  at least 3.2% absolute on cVQA and 6.2% absolute on cIC). This highlights the
  value of retrieval-augmented methods and culturally inclusive benchmarks for
  multimodal understanding.
paper_ai_summary: RAVENEA, a retrieval-augmented benchmark, enhances visual culture understanding in VLMs through culture-focused tasks and outperforms non-augmented models across various metrics.
paper_ai_keywords: vision-language models retrieval-augmented generation RAVENEA culture-focused visual question answering culture-informed image captioning multimodal retrievers culturally inclusive benchmarks multimodal understanding