RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture
  Understanding
As vision-language models (VLMs) become increasingly integrated into daily
life, the need for accurate visual culture understanding is becoming critical.
Yet, these models frequently fall short in interpreting cultural nuances
effectively. Prior work has demonstrated the effectiveness of
retrieval-augmented generation (RAG) in enhancing cultural understanding in
text-only settings, while its application in multimodal scenarios remains
underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented
Visual culturE uNdErstAnding), a new benchmark designed to advance visual
culture understanding through retrieval, focusing on two tasks: culture-focused
visual question answering (cVQA) and culture-informed image captioning (cIC).
RAVENEA extends existing datasets by integrating over 10,000 Wikipedia
documents curated and ranked by human annotators. With RAVENEA, we train and
evaluate seven multimodal retrievers for each image query, and measure the
downstream impact of retrieval-augmented inputs across fourteen
state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented
with culture-aware retrieval, outperform their non-augmented counterparts (by
at least 3.2% absolute on cVQA and 6.2% absolute on cIC). This highlights the
value of retrieval-augmented methods and culturally inclusive benchmarks for
multimodal understanding.
682f38862783867b8e775d27
Jiaang Li
claimed_verified
2025-05-26T08:17:28.360Z
6686658221f096c7fb4e0b24
/avatars/581f5d56b393b95c74f38a06f6bb4f69.svg
Jiaang Li
user
jaagli
682f38862783867b8e775d28
Yifei Yuan
682f38862783867b8e775d29
Wenyan Li
682f38862783867b8e775d2a
Mohammad Aliannejadi
682f38862783867b8e775d2b
Daniel Hershcovich
682f38862783867b8e775d2c
Anders Søgaard
682f38862783867b8e775d2d
Ivan Vulić
682f38862783867b8e775d2e
Wenxuan Zhang
682f38862783867b8e775d2f
Paul Pu Liang
682f38862783867b8e775d30
Yang Deng
682f38862783867b8e775d31
Serge Belongie
RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture
  Understanding
As vision-language models (VLMs) become increasingly integrated into daily
life, the need for accurate visual culture understanding is becoming critical.
Yet, these models frequently fall short in interpreting cultural nuances
effectively. Prior work has demonstrated the effectiveness of
retrieval-augmented generation (RAG) in enhancing cultural understanding in
text-only settings, while its application in multimodal scenarios remains
underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented
Visual culturE uNdErstAnding), a new benchmark designed to advance visual
culture understanding through retrieval, focusing on two tasks: culture-focused
visual question answering (cVQA) and culture-informed image captioning (cIC).
RAVENEA extends existing datasets by integrating over 10,000 Wikipedia
documents curated and ranked by human annotators. With RAVENEA, we train and
evaluate seven multimodal retrievers for each image query, and measure the
downstream impact of retrieval-augmented inputs across fourteen
state-of-the-art VLMs. Our results show that lightweight VLMs, when augmented
with culture-aware retrieval, outperform their non-augmented counterparts (by
at least 3.2% absolute on cVQA and 6.2% absolute on cIC). This highlights the
value of retrieval-augmented methods and culturally inclusive benchmarks for
multimodal understanding.
RAVENEA, a retrieval-augmented benchmark, enhances visual culture understanding in VLMs through culture-focused tasks and outperforms non-augmented models across various metrics.
vision-language models retrieval-augmented generation RAVENEA culture-focused visual question answering culture-informed image captioning multimodal retrievers culturally inclusive benchmarks multimodal understanding