CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
<|reference_start|>CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark: Visual Question Answering (VQA) is an important task in multimodal AI, and it is often used to test the ability of vision-language models to understand and reason on knowledge present in both visual and textual data. However, most of the current VQA models use datasets that are primarily focused on English and a few major world languages, with images that are typically Western-centric. While recent efforts have tried to increase the number of languages covered on VQA datasets, they still lack diversity in low-resource languages. More importantly, although these datasets often extend their linguistic range via translation or some other approaches, they usually keep images the same, resulting in narrow cultural representation. To address these limitations, we construct CVQA, a new Culturally-diverse multilingual Visual Question Answering benchmark, designed to cover a rich set of languages and cultures, where we engage native speakers and cultural experts in the data collection process. As a result, CVQA includes culturally-driven images and questions from across 28 countries on four continents, covering 26 languages with 11 scripts, providing a total of 9k questions. We then benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and show that the dataset is challenging for the current state-of-the-art models. This benchmark can serve as a probing evaluation suite for assessing the cultural capability and bias of multimodal models and hopefully encourage more research efforts toward increasing cultural awareness and linguistic diversity in this field.<|reference_end|>
@article{romero2024cvqa:,
    title={CVQA: Culturally-diverse Multilingual Visual Question Answering
  Benchmark},
    author={David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy
  Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev,
  Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian
  Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio
  Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig
  Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok
  Biadglign Ademtew, Hern'an Maina, Holy Lovenia, Israel Abebe Azime, Jan
  Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas,
  Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas
  Nagasinghe, Luciana Benotti, Luis Fernando D'Haro, Marcelo Viridiano, Marcos
  Estecha-Garitagoitia, Maria Camila Buitrago Cabrera, Mario
  Rodr'iguez-Cantelar, M'elanie Jouitteau, Mihail Mihaylov, Mohamed Fazli
  Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene
  Otgonbold, Naome Etori, Olivier Niyomugisha, Paula M'onica Silva, Pranjal
  Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel
  Cahyawijaya, Santiago G'ongora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki
  Kuribayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent,
  Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei
  Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio,
  Alham Fikri Aji},
    journal={arXiv preprint arXiv:2406.05967},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05967},
    primaryClass={cs.CV cs.AI cs.CL cs.LG}
}