title: MedFuzz: Exploring the Robustness of Large Language Models in Medical
    Question Answering
summary: Large language models (LLM) have achieved impressive performance on medical
  question-answering benchmarks. However, high benchmark accuracy does not imply
  that the performance generalizes to real-world clinical settings. Medical
  question-answering benchmarks rely on assumptions consistent with quantifying
  LLM performance but that may not hold in the open world of the clinic. Yet LLMs
  learn broad knowledge that can help the LLM generalize to practical conditions
  regardless of unrealistic assumptions in celebrated benchmarks. We seek to
  quantify how well LLM medical question-answering benchmark performance
  generalizes when benchmark assumptions are violated. Specifically, we present
  an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz
  attempts to modify benchmark questions in ways aimed at confounding the LLM. We
  demonstrate the approach by targeting strong assumptions about patient
  characteristics presented in the MedQA benchmark. Successful "attacks" modify a
  benchmark item in ways that would be unlikely to fool a medical expert but
  nonetheless "trick" the LLM into changing from a correct to an incorrect
  answer. Further, we present a permutation test technique that can ensure a
  successful attack is statistically significant. We show how to use performance
  on a "MedFuzzed" benchmark, as well as individual successful attacks. The
  methods show promise at providing insights into the ability of an LLM to
  operate robustly in more realistic settings.
paper_authors._id: 666919ea2de144452f920f45
paper_authors.name: Robert Osazuwa Ness
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:47:27.299Z
paper_authors._id: 6384e1e9b2906edaf835dea4
paper_authors.avatarUrl: /avatars/1087a75f7dbfd65b4d18d369d38eb5be.svg
paper_authors.fullname: Robert Osazuwa Ness
paper_authors.type: user
paper_authors.user: osazuwa
paper_authors._id: 666919ea2de144452f920f46
paper_authors.name: Katie Matton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:47:33.622Z
paper_authors._id: 65c04c08262a04f94c4852cd
paper_authors.avatarUrl: /avatars/17f727f8e7e7322a2af571d664ddff65.svg
paper_authors.fullname: Katie Matton
paper_authors.type: user
paper_authors.user: kmatton
paper_authors._id: 666919ea2de144452f920f47
paper_authors.name: Hayden Helm
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:47:39.133Z
paper_authors._id: 651726151ed9cd2cad93106f
paper_authors.avatarUrl: /avatars/79db890fc8615700d9df05b264f3ca86.svg
paper_authors.fullname: Hayden Helm
paper_authors.type: user
paper_authors.user: hshelm
paper_authors._id: 666919ea2de144452f920f48
paper_authors.name: Sheng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:33:59.861Z
paper_authors._id: 6234c11b7d5de9839bc44163
paper_authors.avatarUrl: /avatars/3ca569596f9c7134e8d4b560a06ee1e7.svg
paper_authors.fullname: Sheng Zhang
paper_authors.type: user
paper_authors.user: shengz
paper_authors._id: 666919ea2de144452f920f49
paper_authors.name: Junaid Bajwa
paper_authors._id: 666919ea2de144452f920f4a
paper_authors.name: Carey E. Priebe
paper_authors._id: 666919ea2de144452f920f4b
paper_authors.name: Eric Horvitz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:48:21.339Z
paper_authors._id: 6316576a76c93113fb15ae7f
paper_authors.avatarUrl: /avatars/377045263dca803819289394da719ac9.svg
paper_authors.fullname: Eric Horvitz
paper_authors.type: user
paper_authors.user: skaviouz
paper_title: MedFuzz: Exploring the Robustness of Large Language Models in Medical
    Question Answering
paper_summary: Large language models (LLM) have achieved impressive performance on medical
  question-answering benchmarks. However, high benchmark accuracy does not imply
  that the performance generalizes to real-world clinical settings. Medical
  question-answering benchmarks rely on assumptions consistent with quantifying
  LLM performance but that may not hold in the open world of the clinic. Yet LLMs
  learn broad knowledge that can help the LLM generalize to practical conditions
  regardless of unrealistic assumptions in celebrated benchmarks. We seek to
  quantify how well LLM medical question-answering benchmark performance
  generalizes when benchmark assumptions are violated. Specifically, we present
  an adversarial method that we call MedFuzz (for medical fuzzing). MedFuzz
  attempts to modify benchmark questions in ways aimed at confounding the LLM. We
  demonstrate the approach by targeting strong assumptions about patient
  characteristics presented in the MedQA benchmark. Successful "attacks" modify a
  benchmark item in ways that would be unlikely to fool a medical expert but
  nonetheless "trick" the LLM into changing from a correct to an incorrect
  answer. Further, we present a permutation test technique that can ensure a
  successful attack is statistically significant. We show how to use performance
  on a "MedFuzzed" benchmark, as well as individual successful attacks. The
  methods show promise at providing insights into the ability of an LLM to
  operate robustly in more realistic settings.
paper_ai_keywords: large language models (LLM) medical question-answering benchmarks adversarial method MedFuzz (medical fuzzing) permutation test technique MedQA benchmark