title: WorldMedQA-V: a multilingual, multimodal medical examination dataset for
    multimodal language models evaluation
summary: Multimodal/vision language models (VLMs) are increasingly being deployed in
  healthcare settings worldwide, necessitating robust benchmarks to ensure their
  safety, efficacy, and fairness. Multiple-choice question and answer (QA)
  datasets derived from national medical examinations have long served as
  valuable evaluation tools, but existing datasets are largely text-only and
  available in a limited subset of languages and countries. To address these
  challenges, we present WorldMedQA-V, an updated multilingual, multimodal
  benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V
  includes 568 labeled multiple-choice QAs paired with 568 medical images from
  four countries (Brazil, Israel, Japan, and Spain), covering original languages
  and validated English translations by native clinicians, respectively. Baseline
  performance for common open- and closed-source models are provided in the local
  language and English translations, and with and without images provided to the
  model. The WorldMedQA-V benchmark aims to better match AI systems to the
  diverse healthcare environments in which they are deployed, fostering more
  equitable, effective, and representative applications.
paper_authors._id: 6710cf9452f304c3e265e2ab
paper_authors.name: João Matos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:14:18.443Z
paper_authors._id: 6668600d9a0868622619157f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6668600d9a0868622619157f/Y7Avc94fxlfSiQSweUqvo.jpeg
paper_authors.fullname: João matos
paper_authors.type: user
paper_authors.user: joamats
paper_authors._id: 6710cf9452f304c3e265e2ac
paper_authors.name: Shan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:23:46.339Z
paper_authors._id: 63600b93d9e4214b9a67807c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63600b93d9e4214b9a67807c/BovBE1oiZPdU8Td9I2YIn.jpeg
paper_authors.fullname: Shan Chen
paper_authors.type: user
paper_authors.user: shanchen
paper_authors._id: 6710cf9452f304c3e265e2ad
paper_authors.name: Siena Placino
paper_authors._id: 6710cf9452f304c3e265e2ae
paper_authors.name: Yingya Li
paper_authors._id: 6710cf9452f304c3e265e2af
paper_authors.name: Juan Carlos Climent Pardo
paper_authors._id: 6710cf9452f304c3e265e2b0
paper_authors.name: Daphna Idan
paper_authors._id: 6710cf9452f304c3e265e2b1
paper_authors.name: Takeshi Tohyama
paper_authors._id: 6710cf9452f304c3e265e2b2
paper_authors.name: David Restrepo
paper_authors._id: 6710cf9452f304c3e265e2b3
paper_authors.name: Luis F. Nakayama
paper_authors._id: 6710cf9452f304c3e265e2b4
paper_authors.name: Jose M. M. Pascual-Leone
paper_authors._id: 6710cf9452f304c3e265e2b5
paper_authors.name: Guergana Savova
paper_authors._id: 6710cf9452f304c3e265e2b6
paper_authors.name: Hugo Aerts
paper_authors._id: 6710cf9452f304c3e265e2b7
paper_authors.name: Leo A. Celi
paper_authors._id: 6710cf9452f304c3e265e2b8
paper_authors.name: A. Ian Wong
paper_authors._id: 6710cf9452f304c3e265e2b9
paper_authors.name: Danielle S. Bitterman
paper_authors._id: 6710cf9452f304c3e265e2ba
paper_authors.name: Jack Gallifant
paper_title: WorldMedQA-V: a multilingual, multimodal medical examination dataset for
    multimodal language models evaluation
paper_summary: Multimodal/vision language models (VLMs) are increasingly being deployed in
  healthcare settings worldwide, necessitating robust benchmarks to ensure their
  safety, efficacy, and fairness. Multiple-choice question and answer (QA)
  datasets derived from national medical examinations have long served as
  valuable evaluation tools, but existing datasets are largely text-only and
  available in a limited subset of languages and countries. To address these
  challenges, we present WorldMedQA-V, an updated multilingual, multimodal
  benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V
  includes 568 labeled multiple-choice QAs paired with 568 medical images from
  four countries (Brazil, Israel, Japan, and Spain), covering original languages
  and validated English translations by native clinicians, respectively. Baseline
  performance for common open- and closed-source models are provided in the local
  language and English translations, and with and without images provided to the
  model. The WorldMedQA-V benchmark aims to better match AI systems to the
  diverse healthcare environments in which they are deployed, fostering more
  equitable, effective, and representative applications.
paper_ai_keywords: multimodal/vision language models (VLMs) multiple-choice question and answer (QA) multilingual, multimodal benchmarking dataset worldmedqa-v medical images native clinicians baseline performance closed-source models open-source models