title: CVQA: Culturally-diverse Multilingual Visual Question Answering
    Benchmark
summary: Visual Question Answering (VQA) is an important task in multimodal AI, and it
  is often used to test the ability of vision-language models to understand and
  reason on knowledge present in both visual and textual data. However, most of
  the current VQA models use datasets that are primarily focused on English and a
  few major world languages, with images that are typically Western-centric.
  While recent efforts have tried to increase the number of languages covered on
  VQA datasets, they still lack diversity in low-resource languages. More
  importantly, although these datasets often extend their linguistic range via
  translation or some other approaches, they usually keep images the same,
  resulting in narrow cultural representation. To address these limitations, we
  construct CVQA, a new Culturally-diverse multilingual Visual Question Answering
  benchmark, designed to cover a rich set of languages and cultures, where we
  engage native speakers and cultural experts in the data collection process. As
  a result, CVQA includes culturally-driven images and questions from across 28
  countries on four continents, covering 26 languages with 11 scripts, providing
  a total of 9k questions. We then benchmark several Multimodal Large Language
  Models (MLLMs) on CVQA, and show that the dataset is challenging for the
  current state-of-the-art models. This benchmark can serve as a probing
  evaluation suite for assessing the cultural capability and bias of multimodal
  models and hopefully encourage more research efforts toward increasing cultural
  awareness and linguistic diversity in this field.
paper_authors._id: 666c4afb14f1c262fefaa666
paper_authors.name: David Romero
paper_authors._id: 666c4afb14f1c262fefaa667
paper_authors.name: Chenyang Lyu
paper_authors._id: 666c4afb14f1c262fefaa668
paper_authors.name: Haryo Akbarianto Wibowo
paper_authors._id: 666c4afb14f1c262fefaa669
paper_authors.name: Teresa Lynn
paper_authors._id: 666c4afb14f1c262fefaa66a
paper_authors.name: Injy Hamed
paper_authors._id: 666c4afb14f1c262fefaa66b
paper_authors.name: Aditya Nanda Kishore
paper_authors._id: 666c4afb14f1c262fefaa66c
paper_authors.name: Aishik Mandal
paper_authors._id: 666c4afb14f1c262fefaa66d
paper_authors.name: Alina Dragonetti
paper_authors._id: 666c4afb14f1c262fefaa66e
paper_authors.name: Artem Abzaliev
paper_authors._id: 666c4afb14f1c262fefaa66f
paper_authors.name: Atnafu Lambebo Tonja
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:50:14.586Z
paper_authors._id: 6245232cf59b8d262df30db6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6245232cf59b8d262df30db6/zmQuv1vqGL3MsCAnzGm9y.jpeg
paper_authors.fullname: Atnafu Lambebo Tonja
paper_authors.type: user
paper_authors.user: Atnafu
paper_authors._id: 666c4afb14f1c262fefaa670
paper_authors.name: Bontu Fufa Balcha
paper_authors._id: 666c4afb14f1c262fefaa671
paper_authors.name: Chenxi Whitehouse
paper_authors._id: 666c4afb14f1c262fefaa672
paper_authors.name: Christian Salamea
paper_authors._id: 666c4afb14f1c262fefaa673
paper_authors.name: Dan John Velasco
paper_authors._id: 666c4afb14f1c262fefaa674
paper_authors.name: David Ifeoluwa Adelani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:58:21.285Z
paper_authors._id: 5fcc1929563427b03e9af259
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620642852901-5fcc1929563427b03e9af259.jpeg
paper_authors.fullname: David Adelani
paper_authors.type: user
paper_authors.user: Davlan
paper_authors._id: 666c4afb14f1c262fefaa675
paper_authors.name: David Le Meur
paper_authors._id: 666c4afb14f1c262fefaa676
paper_authors.name: Emilio Villa-Cueva
paper_authors._id: 666c4afb14f1c262fefaa677
paper_authors.name: Fajri Koto
paper_authors._id: 666c4afb14f1c262fefaa678
paper_authors.name: Fauzan Farooqui
paper_authors._id: 666c4afb14f1c262fefaa679
paper_authors.name: Frederico Belcavello
paper_authors._id: 666c4afb14f1c262fefaa67a
paper_authors.name: Ganzorig Batnasan
paper_authors._id: 666c4afb14f1c262fefaa67b
paper_authors.name: Gisela Vallejo
paper_authors._id: 666c4afb14f1c262fefaa67c
paper_authors.name: Grainne Caulfield
paper_authors._id: 666c4afb14f1c262fefaa67d
paper_authors.name: Guido Ivetta
paper_authors._id: 666c4afb14f1c262fefaa67e
paper_authors.name: Haiyue Song
paper_authors._id: 666c4afb14f1c262fefaa67f
paper_authors.name: Henok Biadglign Ademtew
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-12T09:19:37.797Z
paper_authors._id: 62c488530a06c03931610fe0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c488530a06c03931610fe0/Zi5mTvE-R0bHOjuUEl56k.jpeg
paper_authors.fullname: Henok Ademtew
paper_authors.type: user
paper_authors.user: Henok
paper_authors._id: 666c4afb14f1c262fefaa680
paper_authors.name: Hernán Maina
paper_authors._id: 666c4afb14f1c262fefaa681
paper_authors.name: Holy Lovenia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:20:10.842Z
paper_authors._id: 616cd2e040e2f69baa1c7af2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616cd2e040e2f69baa1c7af2/9WbFA6rNfmEh3_hzFFWNJ.png
paper_authors.fullname: Holy Lovenia
paper_authors.type: user
paper_authors.user: holylovenia
paper_authors._id: 666c4afb14f1c262fefaa682
paper_authors.name: Israel Abebe Azime
paper_authors._id: 666c4afb14f1c262fefaa683
paper_authors.name: Jan Christian Blaise Cruz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T16:15:24.872Z
paper_authors._id: 5e89aea283fb7a4fc16792bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e89aea283fb7a4fc16792bf/h_hxnVJdufZHDU8bG-5Db.png
paper_authors.fullname: Jan Christian Blaise Cruz
paper_authors.type: user
paper_authors.user: jcblaise
paper_authors._id: 666c4afb14f1c262fefaa684
paper_authors.name: Jay Gala
paper_authors._id: 666c4afb14f1c262fefaa685
paper_authors.name: Jiahui Geng
paper_authors._id: 666c4afb14f1c262fefaa686
paper_authors.name: Jesus-German Ortiz-Barajas
paper_authors._id: 666c4afb14f1c262fefaa687
paper_authors.name: Jinheon Baek
paper_authors._id: 666c4afb14f1c262fefaa688
paper_authors.name: Jocelyn Dunstan
paper_authors._id: 666c4afb14f1c262fefaa689
paper_authors.name: Laura Alonso Alemany
paper_authors._id: 666c4afb14f1c262fefaa68a
paper_authors.name: Kumaranage Ravindu Yasas Nagasinghe
paper_authors._id: 666c4afb14f1c262fefaa68b
paper_authors.name: Luciana Benotti
paper_authors._id: 666c4afb14f1c262fefaa68c
paper_authors.name: Luis Fernando D'Haro
paper_authors._id: 666c4afb14f1c262fefaa68d
paper_authors.name: Marcelo Viridiano
paper_authors._id: 666c4afb14f1c262fefaa68e
paper_authors.name: Marcos Estecha-Garitagoitia
paper_authors._id: 666c4afb14f1c262fefaa68f
paper_authors.name: Maria Camila Buitrago Cabrera
paper_authors._id: 666c4afb14f1c262fefaa690
paper_authors.name: Mario Rodríguez-Cantelar
paper_authors._id: 666c4afb14f1c262fefaa691
paper_authors.name: Mélanie Jouitteau
paper_authors._id: 666c4afb14f1c262fefaa692
paper_authors.name: Mihail Mihaylov
paper_authors._id: 666c4afb14f1c262fefaa693
paper_authors.name: Mohamed Fazli Mohamed Imam
paper_authors._id: 666c4afb14f1c262fefaa694
paper_authors.name: Muhammad Farid Adilazuarda
paper_authors._id: 666c4afb14f1c262fefaa695
paper_authors.name: Munkhjargal Gochoo
paper_authors._id: 666c4afb14f1c262fefaa696
paper_authors.name: Munkh-Erdene Otgonbold
paper_authors._id: 666c4afb14f1c262fefaa697
paper_authors.name: Naome Etori
paper_authors._id: 666c4afb14f1c262fefaa698
paper_authors.name: Olivier Niyomugisha
paper_authors._id: 666c4afb14f1c262fefaa699
paper_authors.name: Paula Mónica Silva
paper_authors._id: 666c4afb14f1c262fefaa69a
paper_authors.name: Pranjal Chitale
paper_authors._id: 666c4afb14f1c262fefaa69b
paper_authors.name: Raj Dabre
paper_authors._id: 666c4afb14f1c262fefaa69c
paper_authors.name: Rendi Chevi
paper_authors._id: 666c4afb14f1c262fefaa69d
paper_authors.name: Ruochen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T15:22:50.279Z
paper_authors._id: 5fc54c33fa6eef7667a4d699
paper_authors.avatarUrl: /avatars/24cce84812f724fdbcd950bbbae9b154.svg
paper_authors.fullname: Ruochen
paper_authors.type: user
paper_authors.user: ruochenz
paper_authors._id: 666c4afb14f1c262fefaa69e
paper_authors.name: Ryandito Diandaru
paper_authors._id: 666c4afb14f1c262fefaa69f
paper_authors.name: Samuel Cahyawijaya
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:20:07.976Z
paper_authors._id: 5f5c4d49e56d546cd623309b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599884443706-noauth.jpeg
paper_authors.fullname: Samuel Cahyawijaya
paper_authors.type: user
paper_authors.user: samuelcahyawijaya
paper_authors._id: 666c4afb14f1c262fefaa6a0
paper_authors.name: Santiago Góngora
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T07:12:23.015Z
paper_authors._id: 626a938b9873fd89deba7ee0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651151746141-noauth.jpeg
paper_authors.fullname: Santiago Góngora
paper_authors.type: user
paper_authors.user: sgongora27
paper_authors._id: 666c4afb14f1c262fefaa6a1
paper_authors.name: Soyeong Jeong
paper_authors._id: 666c4afb14f1c262fefaa6a2
paper_authors.name: Sukannya Purkayastha
paper_authors._id: 666c4afb14f1c262fefaa6a3
paper_authors.name: Tatsuki Kuribayashi
paper_authors._id: 666c4afb14f1c262fefaa6a4
paper_authors.name: Thanmay Jayakumar
paper_authors._id: 666c4afb14f1c262fefaa6a5
paper_authors.name: Tiago Timponi Torrent
paper_authors._id: 666c4afb14f1c262fefaa6a6
paper_authors.name: Toqeer Ehsan
paper_authors._id: 666c4afb14f1c262fefaa6a7
paper_authors.name: Vladimir Araujo
paper_authors._id: 666c4afb14f1c262fefaa6a8
paper_authors.name: Yova Kementchedjhieva
paper_authors._id: 666c4afb14f1c262fefaa6a9
paper_authors.name: Zara Burzo
paper_authors._id: 666c4afb14f1c262fefaa6aa
paper_authors.name: Zheng Wei Lim
paper_authors._id: 666c4afb14f1c262fefaa6ab
paper_authors.name: Zheng Xin Yong
paper_authors._id: 666c4afb14f1c262fefaa6ac
paper_authors.name: Oana Ignat
paper_authors._id: 666c4afb14f1c262fefaa6ad
paper_authors.name: Joan Nwatu
paper_authors._id: 666c4afb14f1c262fefaa6ae
paper_authors.name: Rada Mihalcea
paper_authors._id: 666c4afb14f1c262fefaa6af
paper_authors.name: Thamar Solorio
paper_authors._id: 666c4afb14f1c262fefaa6b0
paper_authors.name: Alham Fikri Aji
paper_title: CVQA: Culturally-diverse Multilingual Visual Question Answering
    Benchmark
paper_summary: Visual Question Answering (VQA) is an important task in multimodal AI, and it
  is often used to test the ability of vision-language models to understand and
  reason on knowledge present in both visual and textual data. However, most of
  the current VQA models use datasets that are primarily focused on English and a
  few major world languages, with images that are typically Western-centric.
  While recent efforts have tried to increase the number of languages covered on
  VQA datasets, they still lack diversity in low-resource languages. More
  importantly, although these datasets often extend their linguistic range via
  translation or some other approaches, they usually keep images the same,
  resulting in narrow cultural representation. To address these limitations, we
  construct CVQA, a new Culturally-diverse multilingual Visual Question Answering
  benchmark, designed to cover a rich set of languages and cultures, where we
  engage native speakers and cultural experts in the data collection process. As
  a result, CVQA includes culturally-driven images and questions from across 28
  countries on four continents, covering 26 languages with 11 scripts, providing
  a total of 9k questions. We then benchmark several Multimodal Large Language
  Models (MLLMs) on CVQA, and show that the dataset is challenging for the
  current state-of-the-art models. This benchmark can serve as a probing
  evaluation suite for assessing the cultural capability and bias of multimodal
  models and hopefully encourage more research efforts toward increasing cultural
  awareness and linguistic diversity in this field.
paper_ai_keywords: Visual Question Answering (VQA) Culturally-diverse multilingual Visual Question Answering benchmark (CVQA) Multimodal Large Language Models (MLLMs)