title: Towards Expert-Level Medical Question Answering with Large Language
    Models
summary: Recent artificial intelligence (AI) systems have reached milestones in "grand
  challenges" ranging from Go to protein-folding. The capability to retrieve
  medical knowledge, reason over it, and answer medical questions comparably to
  physicians has long been viewed as one such grand challenge.
    Large language models (LLMs) have catalyzed significant progress in medical
  question answering; Med-PaLM was the first model to exceed a "passing" score in
  US Medical Licensing Examination (USMLE) style questions with a score of 67.2%
  on the MedQA dataset. However, this and other prior work suggested significant
  room for improvement, especially when models' answers were compared to
  clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by
  leveraging a combination of base LLM improvements (PaLM 2), medical domain
  finetuning, and prompting strategies including a novel ensemble refinement
  approach.
    Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM
  by over 19% and setting a new state-of-the-art. We also observed performance
  approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU
  clinical topics datasets.
    We performed detailed human evaluations on long-form questions along multiple
  axes relevant to clinical applications. In pairwise comparative ranking of 1066
  consumer medical questions, physicians preferred Med-PaLM 2 answers to those
  produced by physicians on eight of nine axes pertaining to clinical utility (p
  < 0.001). We also observed significant improvements compared to Med-PaLM on
  every evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form
  "adversarial" questions to probe LLM limitations.
    While further studies are necessary to validate the efficacy of these models
  in real-world settings, these results highlight rapid progress towards
  physician-level performance in medical question answering.
paper_authors._id: 64642c4c7572c66a8e649948
paper_authors.name: Karan Singhal
paper_authors._id: 64642c4c7572c66a8e649949
paper_authors.name: Tao Tu
paper_authors._id: 64642c4c7572c66a8e64994a
paper_authors.name: Juraj Gottweis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:21:14.750Z
paper_authors._id: 6381e100cae34727b9ca33ce
paper_authors.avatarUrl: /avatars/f4e047e82e41ec507c6f51aef7560d87.svg
paper_authors.fullname: Juraj Gottweis
paper_authors.type: user
paper_authors.user: juro
paper_authors._id: 64642c4c7572c66a8e64994b
paper_authors.name: Rory Sayres
paper_authors._id: 64642c4c7572c66a8e64994c
paper_authors.name: Ellery Wulczyn
paper_authors._id: 64642c4c7572c66a8e64994d
paper_authors.name: Le Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-19T10:28:09.156Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_authors._id: 64642c4c7572c66a8e64994e
paper_authors.name: Kevin Clark
paper_authors._id: 64642c4c7572c66a8e64994f
paper_authors.name: Stephen Pfohl
paper_authors._id: 64642c4c7572c66a8e649950
paper_authors.name: Heather Cole-Lewis
paper_authors._id: 64642c4c7572c66a8e649951
paper_authors.name: Darlene Neal
paper_authors._id: 64642c4c7572c66a8e649952
paper_authors.name: Mike Schaekermann
paper_authors._id: 64642c4c7572c66a8e649953
paper_authors.name: Amy Wang
paper_authors._id: 64642c4c7572c66a8e649954
paper_authors.name: Mohamed Amin
paper_authors._id: 64642c4c7572c66a8e649955
paper_authors.name: Sami Lachgar
paper_authors._id: 64642c4c7572c66a8e649956
paper_authors.name: Philip Mansfield
paper_authors._id: 64642c4c7572c66a8e649957
paper_authors.name: Sushant Prakash
paper_authors._id: 64642c4c7572c66a8e649958
paper_authors.name: Bradley Green
paper_authors._id: 64642c4c7572c66a8e649959
paper_authors.name: Ewa Dominowska
paper_authors._id: 64642c4c7572c66a8e64995a
paper_authors.name: Blaise Aguera y Arcas
paper_authors._id: 64642c4c7572c66a8e64995b
paper_authors.name: Nenad Tomasev
paper_authors._id: 64642c4c7572c66a8e64995c
paper_authors.name: Yun Liu
paper_authors._id: 64642c4c7572c66a8e64995d
paper_authors.name: Renee Wong
paper_authors._id: 64642c4c7572c66a8e64995e
paper_authors.name: Christopher Semturs
paper_authors._id: 64642c4c7572c66a8e64995f
paper_authors.name: S. Sara Mahdavi
paper_authors._id: 64642c4c7572c66a8e649960
paper_authors.name: Joelle Barral
paper_authors._id: 64642c4c7572c66a8e649961
paper_authors.name: Dale Webster
paper_authors._id: 64642c4c7572c66a8e649962
paper_authors.name: Greg S. Corrado
paper_authors._id: 64642c4c7572c66a8e649963
paper_authors.name: Yossi Matias
paper_authors._id: 64642c4c7572c66a8e649964
paper_authors.name: Shekoofeh Azizi
paper_authors._id: 64642c4c7572c66a8e649965
paper_authors.name: Alan Karthikesalingam
paper_authors._id: 64642c4c7572c66a8e649966
paper_authors.name: Vivek Natarajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T09:24:24.776Z
paper_authors._id: 617a6af2941993035fbaf29c
paper_authors.avatarUrl: /avatars/a389671494782f06ff8502c934090cc0.svg
paper_authors.fullname: Vivek Natarajan
paper_authors.type: user
paper_authors.user: vivnat
paper_title: Towards Expert-Level Medical Question Answering with Large Language
    Models
paper_summary: Recent artificial intelligence (AI) systems have reached milestones in "grand
  challenges" ranging from Go to protein-folding. The capability to retrieve
  medical knowledge, reason over it, and answer medical questions comparably to
  physicians has long been viewed as one such grand challenge.
    Large language models (LLMs) have catalyzed significant progress in medical
  question answering; Med-PaLM was the first model to exceed a "passing" score in
  US Medical Licensing Examination (USMLE) style questions with a score of 67.2%
  on the MedQA dataset. However, this and other prior work suggested significant
  room for improvement, especially when models' answers were compared to
  clinicians' answers. Here we present Med-PaLM 2, which bridges these gaps by
  leveraging a combination of base LLM improvements (PaLM 2), medical domain
  finetuning, and prompting strategies including a novel ensemble refinement
  approach.
    Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLM
  by over 19% and setting a new state-of-the-art. We also observed performance
  approaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLU
  clinical topics datasets.
    We performed detailed human evaluations on long-form questions along multiple
  axes relevant to clinical applications. In pairwise comparative ranking of 1066
  consumer medical questions, physicians preferred Med-PaLM 2 answers to those
  produced by physicians on eight of nine axes pertaining to clinical utility (p
  < 0.001). We also observed significant improvements compared to Med-PaLM on
  every evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form
  "adversarial" questions to probe LLM limitations.
    While further studies are necessary to validate the efficacy of these models
  in real-world settings, these results highlight rapid progress towards
  physician-level performance in medical question answering.
paper_ai_keywords: large language models (LLMs) Med-PaLM US Medical Licensing Examination (USMLE) MedQA dataset Med-PaLM 2 PaLM 2 medical domain finetuning prompting strategies ensemble refinement approach MedMCQA PubMedQA MMLU clinical topics datasets human evaluations clinical utility adversarial questions