AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering
  Benchmark Dataset
Recent advancements in large language model(LLM) performance on medical
multiple choice question (MCQ) benchmarks have stimulated interest from
healthcare providers and patients globally. Particularly in low-and
middle-income countries (LMICs) facing acute physician shortages and lack of
specialists, LLMs offer a potentially scalable pathway to enhance healthcare
access and reduce costs. However, their effectiveness in the Global South,
especially across the African continent, remains to be established. In this
work, we introduce AfriMed-QA, the first large scale Pan-African English
multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open
and closed-ended) sourced from over 60 medical schools across 16 countries,
covering 32 medical specialties. We further evaluate 30 LLMs across multiple
axes including correctness and demographic bias. Our findings show significant
performance variation across specialties and geographies, MCQ performance
clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general
models and smaller edge-friendly LLMs struggle to achieve a passing score.
Interestingly, human evaluations show a consistent consumer preference for LLM
answers and explanations when compared with clinician answers.
6749fecca0b9dfa2dadfedb5
Tobi Olatunji
extracted_confirmed
2024-11-29T18:55:19.987Z
63d78daddb69ae9b9be829ad
/avatars/6543bfe30f2b385f27955743ce79dd7b.svg
Tobi Olatunji
user
tobiolatunji
6749fecca0b9dfa2dadfedb6
Charles Nimo
claimed_verified
2025-07-16T08:16:38.181Z
651f0c208ed730cc006f3c6a
/avatars/46f9ddb62127432e4fd70f7a5c5a934d.svg
Charles Nimo
user
cnimo10
6749fecca0b9dfa2dadfedb7
Abraham Owodunni
claimed_verified
2024-11-29T21:10:07.233Z
626d1e1e72169e781945bf44
/avatars/6bf9f35042b6d939f2ab525816ad0423.svg
Abraham  Owodunni
user
Owos
6749fecca0b9dfa2dadfedb8
Tassallah Abdullahi
claimed_verified
2025-05-26T08:44:54.378Z
66197708b6998e8c45988fff
/avatars/3cafcdbbcb6a10720bcb1e5e0fcf1f4e.svg
Tassallah Abdullahi
user
Tassy24
6749fecca0b9dfa2dadfedb9
Emmanuel Ayodele
6749fecca0b9dfa2dadfedba
Mardhiyah Sanni
6749fecca0b9dfa2dadfedbb
Chinemelu Aka
6749fecca0b9dfa2dadfedbc
Folafunmi Omofoye
6749fecca0b9dfa2dadfedbd
Foutse Yuehgoh
6749fecca0b9dfa2dadfedbe
Timothy Faniran
6749fecca0b9dfa2dadfedbf
Bonaventure F. P. Dossou
6749fecca0b9dfa2dadfedc0
Moshood Yekini
6749fecca0b9dfa2dadfedc1
Jonas Kemp
6749fecca0b9dfa2dadfedc2
Katherine Heller
6749fecca0b9dfa2dadfedc3
Jude Chidubem Omeke
6749fecca0b9dfa2dadfedc4
Chidi Asuzu MD
6749fecca0b9dfa2dadfedc5
Naome A. Etori
6749fecca0b9dfa2dadfedc6
Aimérou Ndiaye
6749fecca0b9dfa2dadfedc7
Ifeoma Okoh
6749fecca0b9dfa2dadfedc8
Evans Doe Ocansey
6749fecca0b9dfa2dadfedc9
Wendy Kinara
6749fecca0b9dfa2dadfedca
Michael Best
6749fecca0b9dfa2dadfedcb
Irfan Essa
6749fecca0b9dfa2dadfedcc
Stephen Edward Moore
6749fecca0b9dfa2dadfedcd
Chris Fourie
6749fecca0b9dfa2dadfedce
Mercy Nyamewaa Asiedu
AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering
  Benchmark Dataset
Recent advancements in large language model(LLM) performance on medical
multiple choice question (MCQ) benchmarks have stimulated interest from
healthcare providers and patients globally. Particularly in low-and
middle-income countries (LMICs) facing acute physician shortages and lack of
specialists, LLMs offer a potentially scalable pathway to enhance healthcare
access and reduce costs. However, their effectiveness in the Global South,
especially across the African continent, remains to be established. In this
work, we introduce AfriMed-QA, the first large scale Pan-African English
multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open
and closed-ended) sourced from over 60 medical schools across 16 countries,
covering 32 medical specialties. We further evaluate 30 LLMs across multiple
axes including correctness and demographic bias. Our findings show significant
performance variation across specialties and geographies, MCQ performance
clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general
models and smaller edge-friendly LLMs struggle to achieve a passing score.
Interestingly, human evaluations show a consistent consumer preference for LLM
answers and explanations when compared with clinician answers.
AfriMed-QA, a large Pan-African medical QA dataset, evaluates various LLMs for medical MCQs, revealing performance variations and consumer preferences.
large language model LLM medical multiple choice question MCQ low-and middle-income countries LMICs physician shortages healthcare access AfriMed-QA medical Question-Answering QA USMLE MedQA biomedical LLMs edge-friendly LLMs human evaluations