CVQA: Culturally-diverse Multilingual Visual Question Answering
  Benchmark
Visual Question Answering (VQA) is an important task in multimodal AI, and it
is often used to test the ability of vision-language models to understand and
reason on knowledge present in both visual and textual data. However, most of
the current VQA models use datasets that are primarily focused on English and a
few major world languages, with images that are typically Western-centric.
While recent efforts have tried to increase the number of languages covered on
VQA datasets, they still lack diversity in low-resource languages. More
importantly, although these datasets often extend their linguistic range via
translation or some other approaches, they usually keep images the same,
resulting in narrow cultural representation. To address these limitations, we
construct CVQA, a new Culturally-diverse multilingual Visual Question Answering
benchmark, designed to cover a rich set of languages and cultures, where we
engage native speakers and cultural experts in the data collection process. As
a result, CVQA includes culturally-driven images and questions from across 28
countries on four continents, covering 26 languages with 11 scripts, providing
a total of 9k questions. We then benchmark several Multimodal Large Language
Models (MLLMs) on CVQA, and show that the dataset is challenging for the
current state-of-the-art models. This benchmark can serve as a probing
evaluation suite for assessing the cultural capability and bias of multimodal
models and hopefully encourage more research efforts toward increasing cultural
awareness and linguistic diversity in this field.
666c4afb14f1c262fefaa666
David Romero
666c4afb14f1c262fefaa667
Chenyang Lyu
claimed_verified
2025-04-23T08:40:32.067Z
6527d8b077bceabaab382a75
/avatars/69caacf9153dbf6a3796693a968b363f.svg
Chenyang Lyu
user
ChenyangLyu
666c4afb14f1c262fefaa668
Haryo Akbarianto Wibowo
666c4afb14f1c262fefaa669
Teresa Lynn
666c4afb14f1c262fefaa66a
Injy Hamed
666c4afb14f1c262fefaa66b
Aditya Nanda Kishore
666c4afb14f1c262fefaa66c
Aishik Mandal
666c4afb14f1c262fefaa66d
Alina Dragonetti
666c4afb14f1c262fefaa66e
Artem Abzaliev
666c4afb14f1c262fefaa66f
Atnafu Lambebo Tonja
claimed_verified
2024-09-03T07:50:14.586Z
6245232cf59b8d262df30db6
https://cdn-avatars.huggingface.co/v1/production/uploads/6245232cf59b8d262df30db6/zmQuv1vqGL3MsCAnzGm9y.jpeg
Atnafu Lambebo Tonja
user
Atnafu
666c4afb14f1c262fefaa670
Bontu Fufa Balcha
666c4afb14f1c262fefaa671
Chenxi Whitehouse
666c4afb14f1c262fefaa672
Christian Salamea
666c4afb14f1c262fefaa673
Dan John Velasco
666c4afb14f1c262fefaa674
David Ifeoluwa Adelani
claimed_verified
2024-12-06T15:58:21.285Z
5fcc1929563427b03e9af259
https://cdn-avatars.huggingface.co/v1/production/uploads/1620642852901-5fcc1929563427b03e9af259.jpeg
David Adelani
user
Davlan
666c4afb14f1c262fefaa675
David Le Meur
666c4afb14f1c262fefaa676
Emilio Villa-Cueva
666c4afb14f1c262fefaa677
Fajri Koto
666c4afb14f1c262fefaa678
Fauzan Farooqui
666c4afb14f1c262fefaa679
Frederico Belcavello
666c4afb14f1c262fefaa67a
Ganzorig Batnasan
666c4afb14f1c262fefaa67b
Gisela Vallejo
666c4afb14f1c262fefaa67c
Grainne Caulfield
666c4afb14f1c262fefaa67d
Guido Ivetta
666c4afb14f1c262fefaa67e
Haiyue Song
666c4afb14f1c262fefaa67f
Henok Biadglign Ademtew
claimed_verified
2025-02-12T09:19:37.797Z
62c488530a06c03931610fe0
https://cdn-avatars.huggingface.co/v1/production/uploads/62c488530a06c03931610fe0/Zi5mTvE-R0bHOjuUEl56k.jpeg
Henok Ademtew
user
Henok
666c4afb14f1c262fefaa680
Hernán Maina
666c4afb14f1c262fefaa681
Holy Lovenia
claimed_verified
2024-06-18T07:20:10.842Z
616cd2e040e2f69baa1c7af2
https://cdn-avatars.huggingface.co/v1/production/uploads/616cd2e040e2f69baa1c7af2/9WbFA6rNfmEh3_hzFFWNJ.png
Holy Lovenia
user
holylovenia
666c4afb14f1c262fefaa682
Israel Abebe Azime
claimed_verified
2025-05-14T07:38:05.293Z
603581672b01f31f03359c99
https://cdn-avatars.huggingface.co/v1/production/uploads/603581672b01f31f03359c99/UQdE-Z5Ua2yF2q6SaLa1z.jpeg
Israel Abebe Azime
user
israel
666c4afb14f1c262fefaa683
Jan Christian Blaise Cruz
claimed_verified
2024-06-17T16:15:24.872Z
5e89aea283fb7a4fc16792bf
https://cdn-avatars.huggingface.co/v1/production/uploads/5e89aea283fb7a4fc16792bf/h_hxnVJdufZHDU8bG-5Db.png
Jan Christian Blaise Cruz
user
jcblaise
666c4afb14f1c262fefaa684
Jay Gala
666c4afb14f1c262fefaa685
Jiahui Geng
666c4afb14f1c262fefaa686
Jesus-German Ortiz-Barajas
666c4afb14f1c262fefaa687
Jinheon Baek
claimed_verified
2025-04-25T08:37:03.406Z
63036b6c5c70c21d0ea79d48
/avatars/a7eb03f5cbd4eaa09fe807bbed8bc0f7.svg
Jinheon Baek
user
jinheon
666c4afb14f1c262fefaa688
Jocelyn Dunstan
666c4afb14f1c262fefaa689
Laura Alonso Alemany
666c4afb14f1c262fefaa68a
Kumaranage Ravindu Yasas Nagasinghe
666c4afb14f1c262fefaa68b
Luciana Benotti
666c4afb14f1c262fefaa68c
Luis Fernando D'Haro
666c4afb14f1c262fefaa68d
Marcelo Viridiano
666c4afb14f1c262fefaa68e
Marcos Estecha-Garitagoitia
666c4afb14f1c262fefaa68f
Maria Camila Buitrago Cabrera
666c4afb14f1c262fefaa690
Mario Rodríguez-Cantelar
666c4afb14f1c262fefaa691
Mélanie Jouitteau
666c4afb14f1c262fefaa692
Mihail Mihaylov
666c4afb14f1c262fefaa693
Mohamed Fazli Mohamed Imam
666c4afb14f1c262fefaa694
Muhammad Farid Adilazuarda
666c4afb14f1c262fefaa695
Munkhjargal Gochoo
666c4afb14f1c262fefaa696
Munkh-Erdene Otgonbold
666c4afb14f1c262fefaa697
Naome Etori
666c4afb14f1c262fefaa698
Olivier Niyomugisha
666c4afb14f1c262fefaa699
Paula Mónica Silva
666c4afb14f1c262fefaa69a
Pranjal Chitale
666c4afb14f1c262fefaa69b
Raj Dabre
666c4afb14f1c262fefaa69c
Rendi Chevi
666c4afb14f1c262fefaa69d
Ruochen Zhang
claimed_verified
2025-03-12T15:22:50.279Z
5fc54c33fa6eef7667a4d699
/avatars/24cce84812f724fdbcd950bbbae9b154.svg
Ruochen
user
ruochenz
666c4afb14f1c262fefaa69e
Ryandito Diandaru
666c4afb14f1c262fefaa69f
Samuel Cahyawijaya
claimed_verified
2024-06-18T07:20:07.976Z
5f5c4d49e56d546cd623309b
https://cdn-avatars.huggingface.co/v1/production/uploads/1599884443706-noauth.jpeg
Samuel Cahyawijaya
user
samuelcahyawijaya
666c4afb14f1c262fefaa6a0
Santiago Góngora
claimed_verified
2024-09-17T07:12:23.015Z
626a938b9873fd89deba7ee0
https://cdn-avatars.huggingface.co/v1/production/uploads/1651151746141-noauth.jpeg
Santiago Góngora
user
sgongora27
666c4afb14f1c262fefaa6a1
Soyeong Jeong
666c4afb14f1c262fefaa6a2
Sukannya Purkayastha
claimed_verified
2025-04-16T11:58:05.935Z
6408e9f2bf3e9a4bb2e09400
https://cdn-avatars.huggingface.co/v1/production/uploads/1678306212334-6408e9f2bf3e9a4bb2e09400.jpeg
Sukannya Purkayastha
user
sukannya
666c4afb14f1c262fefaa6a3
Tatsuki Kuribayashi
666c4afb14f1c262fefaa6a4
Thanmay Jayakumar
666c4afb14f1c262fefaa6a5
Tiago Timponi Torrent
666c4afb14f1c262fefaa6a6
Toqeer Ehsan
666c4afb14f1c262fefaa6a7
Vladimir Araujo
666c4afb14f1c262fefaa6a8
Yova Kementchedjhieva
666c4afb14f1c262fefaa6a9
Zara Burzo
666c4afb14f1c262fefaa6aa
Zheng Wei Lim
666c4afb14f1c262fefaa6ab
Zheng Xin Yong
666c4afb14f1c262fefaa6ac
Oana Ignat
666c4afb14f1c262fefaa6ad
Joan Nwatu
666c4afb14f1c262fefaa6ae
Rada Mihalcea
666c4afb14f1c262fefaa6af
Thamar Solorio
666c4afb14f1c262fefaa6b0
Alham Fikri Aji
CVQA: Culturally-diverse Multilingual Visual Question Answering
  Benchmark
Visual Question Answering (VQA) is an important task in multimodal AI, and it
is often used to test the ability of vision-language models to understand and
reason on knowledge present in both visual and textual data. However, most of
the current VQA models use datasets that are primarily focused on English and a
few major world languages, with images that are typically Western-centric.
While recent efforts have tried to increase the number of languages covered on
VQA datasets, they still lack diversity in low-resource languages. More
importantly, although these datasets often extend their linguistic range via
translation or some other approaches, they usually keep images the same,
resulting in narrow cultural representation. To address these limitations, we
construct CVQA, a new Culturally-diverse multilingual Visual Question Answering
benchmark, designed to cover a rich set of languages and cultures, where we
engage native speakers and cultural experts in the data collection process. As
a result, CVQA includes culturally-driven images and questions from across 28
countries on four continents, covering 26 languages with 11 scripts, providing
a total of 9k questions. We then benchmark several Multimodal Large Language
Models (MLLMs) on CVQA, and show that the dataset is challenging for the
current state-of-the-art models. This benchmark can serve as a probing
evaluation suite for assessing the cultural capability and bias of multimodal
models and hopefully encourage more research efforts toward increasing cultural
awareness and linguistic diversity in this field.
Culturally-diverse multilingual Visual Question Answering benchmark (CVQA) covers 28 countries, 26 languages, and 11 scripts, testing the cultural capability and linguistic diversity of multimodal models.
Visual Question Answering (VQA) multimodal AI vision-language models CVQA Culturally-diverse multilingual Visual Question Answering benchmark Multimodal Large Language Models (MLLMs) cultural capability linguistic diversity