title: AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering
    Benchmark Dataset
summary: Recent advancements in large language model(LLM) performance on medical
  multiple choice question (MCQ) benchmarks have stimulated interest from
  healthcare providers and patients globally. Particularly in low-and
  middle-income countries (LMICs) facing acute physician shortages and lack of
  specialists, LLMs offer a potentially scalable pathway to enhance healthcare
  access and reduce costs. However, their effectiveness in the Global South,
  especially across the African continent, remains to be established. In this
  work, we introduce AfriMed-QA, the first large scale Pan-African English
  multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open
  and closed-ended) sourced from over 60 medical schools across 16 countries,
  covering 32 medical specialties. We further evaluate 30 LLMs across multiple
  axes including correctness and demographic bias. Our findings show significant
  performance variation across specialties and geographies, MCQ performance
  clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general
  models and smaller edge-friendly LLMs struggle to achieve a passing score.
  Interestingly, human evaluations show a consistent consumer preference for LLM
  answers and explanations when compared with clinician answers.
paper_authors._id: 6749fecca0b9dfa2dadfedb5
paper_authors.name: Tobi Olatunji
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-11-29T18:55:19.987Z
paper_authors._id: 63d78daddb69ae9b9be829ad
paper_authors.avatarUrl: /avatars/6543bfe30f2b385f27955743ce79dd7b.svg
paper_authors.fullname: Tobi Olatunji
paper_authors.type: user
paper_authors.user: tobiolatunji
paper_authors._id: 6749fecca0b9dfa2dadfedb6
paper_authors.name: Charles Nimo
paper_authors._id: 6749fecca0b9dfa2dadfedb7
paper_authors.name: Abraham Owodunni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-29T21:10:07.233Z
paper_authors._id: 626d1e1e72169e781945bf44
paper_authors.avatarUrl: /avatars/6bf9f35042b6d939f2ab525816ad0423.svg
paper_authors.fullname: Abraham  Owodunni
paper_authors.type: user
paper_authors.user: Owos
paper_authors._id: 6749fecca0b9dfa2dadfedb8
paper_authors.name: Tassallah Abdullahi
paper_authors._id: 6749fecca0b9dfa2dadfedb9
paper_authors.name: Emmanuel Ayodele
paper_authors._id: 6749fecca0b9dfa2dadfedba
paper_authors.name: Mardhiyah Sanni
paper_authors._id: 6749fecca0b9dfa2dadfedbb
paper_authors.name: Chinemelu Aka
paper_authors._id: 6749fecca0b9dfa2dadfedbc
paper_authors.name: Folafunmi Omofoye
paper_authors._id: 6749fecca0b9dfa2dadfedbd
paper_authors.name: Foutse Yuehgoh
paper_authors._id: 6749fecca0b9dfa2dadfedbe
paper_authors.name: Timothy Faniran
paper_authors._id: 6749fecca0b9dfa2dadfedbf
paper_authors.name: Bonaventure F. P. Dossou
paper_authors._id: 6749fecca0b9dfa2dadfedc0
paper_authors.name: Moshood Yekini
paper_authors._id: 6749fecca0b9dfa2dadfedc1
paper_authors.name: Jonas Kemp
paper_authors._id: 6749fecca0b9dfa2dadfedc2
paper_authors.name: Katherine Heller
paper_authors._id: 6749fecca0b9dfa2dadfedc3
paper_authors.name: Jude Chidubem Omeke
paper_authors._id: 6749fecca0b9dfa2dadfedc4
paper_authors.name: Chidi Asuzu MD
paper_authors._id: 6749fecca0b9dfa2dadfedc5
paper_authors.name: Naome A. Etori
paper_authors._id: 6749fecca0b9dfa2dadfedc6
paper_authors.name: Aimérou Ndiaye
paper_authors._id: 6749fecca0b9dfa2dadfedc7
paper_authors.name: Ifeoma Okoh
paper_authors._id: 6749fecca0b9dfa2dadfedc8
paper_authors.name: Evans Doe Ocansey
paper_authors._id: 6749fecca0b9dfa2dadfedc9
paper_authors.name: Wendy Kinara
paper_authors._id: 6749fecca0b9dfa2dadfedca
paper_authors.name: Michael Best
paper_authors._id: 6749fecca0b9dfa2dadfedcb
paper_authors.name: Irfan Essa
paper_authors._id: 6749fecca0b9dfa2dadfedcc
paper_authors.name: Stephen Edward Moore
paper_authors._id: 6749fecca0b9dfa2dadfedcd
paper_authors.name: Chris Fourie
paper_authors._id: 6749fecca0b9dfa2dadfedce
paper_authors.name: Mercy Nyamewaa Asiedu
paper_title: AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering
    Benchmark Dataset
paper_summary: Recent advancements in large language model(LLM) performance on medical
  multiple choice question (MCQ) benchmarks have stimulated interest from
  healthcare providers and patients globally. Particularly in low-and
  middle-income countries (LMICs) facing acute physician shortages and lack of
  specialists, LLMs offer a potentially scalable pathway to enhance healthcare
  access and reduce costs. However, their effectiveness in the Global South,
  especially across the African continent, remains to be established. In this
  work, we introduce AfriMed-QA, the first large scale Pan-African English
  multi-specialty medical Question-Answering (QA) dataset, 15,000 questions (open
  and closed-ended) sourced from over 60 medical schools across 16 countries,
  covering 32 medical specialties. We further evaluate 30 LLMs across multiple
  axes including correctness and demographic bias. Our findings show significant
  performance variation across specialties and geographies, MCQ performance
  clearly lags USMLE (MedQA). We find that biomedical LLMs underperform general
  models and smaller edge-friendly LLMs struggle to achieve a passing score.
  Interestingly, human evaluations show a consistent consumer preference for LLM
  answers and explanations when compared with clinician answers.
paper_ai_keywords: large language model (LLM) multiple choice question (MCQ) benchmarks healthcare providers patients low-and middle-income countries (LMICs) physician shortages specialists scalable pathway healthcare access costs AfriMed-QA Pan-African English multi-specialty Question-Answering (QA) dataset medical schools specialties performance variation geographies USMLE (MedQA) biomedical LLMs general models edge-friendly LLMs passing score human evaluations consumer preference clinician answers