title: Towards Accurate Differential Diagnosis with Large Language Models
summary: An accurate differential diagnosis (DDx) is a cornerstone of medical care,
  often reached through an iterative process of interpretation that combines
  clinical history, physical examination, investigations and procedures.
  Interactive interfaces powered by Large Language Models (LLMs) present new
  opportunities to both assist and automate aspects of this process. In this
  study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its
  ability to generate a DDx alone or as an aid to clinicians. 20 clinicians
  evaluated 302 challenging, real-world medical cases sourced from the New
  England Journal of Medicine (NEJM) case reports. Each case report was read by
  two clinicians, who were randomized to one of two assistive conditions: either
  assistance from search engines and standard medical resources, or LLM
  assistance in addition to these tools. All clinicians provided a baseline,
  unassisted DDx prior to using the respective assistive tools. Our LLM for DDx
  exhibited standalone performance that exceeded that of unassisted clinicians
  (top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study
  arms, the DDx quality score was higher for clinicians assisted by our LLM
  (top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%)
  (McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p =
  0.03). Further, clinicians assisted by our LLM arrived at more comprehensive
  differential lists than those without its assistance. Our study suggests that
  our LLM for DDx has potential to improve clinicians' diagnostic reasoning and
  accuracy in challenging cases, meriting further real-world evaluation for its
  ability to empower physicians and widen patients' access to specialist-level
  expertise.
paper_authors._id: 656d32393dbac3a83d399ada
paper_authors.name: Daniel McDuff
paper_authors._id: 656d32393dbac3a83d399adb
paper_authors.name: Mike Schaekermann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:05:32.096Z
paper_authors._id: 64ebb75f171e6c9862d261c9
paper_authors.avatarUrl: /avatars/b7750f5d6281f906cfa5c717a44e2e13.svg
paper_authors.fullname: Mike Schaekermann
paper_authors.type: user
paper_authors.user: mikeschaekermann
paper_authors._id: 656d32393dbac3a83d399adc
paper_authors.name: Tao Tu
paper_authors._id: 656d32393dbac3a83d399add
paper_authors.name: Anil Palepu
paper_authors._id: 656d32393dbac3a83d399ade
paper_authors.name: Amy Wang
paper_authors._id: 656d32393dbac3a83d399adf
paper_authors.name: Jake Garrison
paper_authors._id: 656d32393dbac3a83d399ae0
paper_authors.name: Karan Singhal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:06:14.681Z
paper_authors._id: 654bb16a48b474120258371c
paper_authors.avatarUrl: /avatars/7ad4a33503533ddbb54c71b634fda89b.svg
paper_authors.fullname: Karan Singhal
paper_authors.type: user
paper_authors.user: karan1149
paper_authors._id: 656d32393dbac3a83d399ae1
paper_authors.name: Yash Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-12T12:06:36.684Z
paper_authors._id: 65a3178a534e60db99ec2cfc
paper_authors.avatarUrl: /avatars/f0c5c364d8b2d1bdf7451973014f13be.svg
paper_authors.fullname: Yash Sharma
paper_authors.type: user
paper_authors.user: yashrma
paper_authors._id: 656d32393dbac3a83d399ae2
paper_authors.name: Shekoofeh Azizi
paper_authors._id: 656d32393dbac3a83d399ae3
paper_authors.name: Kavita Kulkarni
paper_authors._id: 656d32393dbac3a83d399ae4
paper_authors.name: Le Hou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:07:00.306Z
paper_authors._id: 626b2402c804c432c1b5004c
paper_authors.avatarUrl: /avatars/a119644422335e260af0ca4ef6399870.svg
paper_authors.fullname: Le Hou
paper_authors.type: user
paper_authors.user: solooobear
paper_authors._id: 656d32393dbac3a83d399ae5
paper_authors.name: Yong Cheng
paper_authors._id: 656d32393dbac3a83d399ae6
paper_authors.name: Yun Liu
paper_authors._id: 656d32393dbac3a83d399ae7
paper_authors.name: S Sara Mahdavi
paper_authors._id: 656d32393dbac3a83d399ae8
paper_authors.name: Sushant Prakash
paper_authors._id: 656d32393dbac3a83d399ae9
paper_authors.name: Anupam Pathak
paper_authors._id: 656d32393dbac3a83d399aea
paper_authors.name: Christopher Semturs
paper_authors._id: 656d32393dbac3a83d399aeb
paper_authors.name: Shwetak Patel
paper_authors._id: 656d32393dbac3a83d399aec
paper_authors.name: Dale R Webster
paper_authors._id: 656d32393dbac3a83d399aed
paper_authors.name: Ewa Dominowska
paper_authors._id: 656d32393dbac3a83d399aee
paper_authors.name: Juraj Gottweis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:08:46.204Z
paper_authors._id: 6381e100cae34727b9ca33ce
paper_authors.avatarUrl: /avatars/f4e047e82e41ec507c6f51aef7560d87.svg
paper_authors.fullname: Juraj Gottweis
paper_authors.type: user
paper_authors.user: juro
paper_authors._id: 656d32393dbac3a83d399aef
paper_authors.name: Joelle Barral
paper_authors._id: 656d32393dbac3a83d399af0
paper_authors.name: Katherine Chou
paper_authors._id: 656d32393dbac3a83d399af1
paper_authors.name: Greg S Corrado
paper_authors._id: 656d32393dbac3a83d399af2
paper_authors.name: Yossi Matias
paper_authors._id: 656d32393dbac3a83d399af3
paper_authors.name: Jake Sunshine
paper_authors._id: 656d32393dbac3a83d399af4
paper_authors.name: Alan Karthikesalingam
paper_authors._id: 656d32393dbac3a83d399af5
paper_authors.name: Vivek Natarajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:08:01.730Z
paper_authors._id: 617a6af2941993035fbaf29c
paper_authors.avatarUrl: /avatars/a389671494782f06ff8502c934090cc0.svg
paper_authors.fullname: Vivek Natarajan
paper_authors.type: user
paper_authors.user: vivnat
paper_title: Towards Accurate Differential Diagnosis with Large Language Models
paper_summary: An accurate differential diagnosis (DDx) is a cornerstone of medical care,
  often reached through an iterative process of interpretation that combines
  clinical history, physical examination, investigations and procedures.
  Interactive interfaces powered by Large Language Models (LLMs) present new
  opportunities to both assist and automate aspects of this process. In this
  study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its
  ability to generate a DDx alone or as an aid to clinicians. 20 clinicians
  evaluated 302 challenging, real-world medical cases sourced from the New
  England Journal of Medicine (NEJM) case reports. Each case report was read by
  two clinicians, who were randomized to one of two assistive conditions: either
  assistance from search engines and standard medical resources, or LLM
  assistance in addition to these tools. All clinicians provided a baseline,
  unassisted DDx prior to using the respective assistive tools. Our LLM for DDx
  exhibited standalone performance that exceeded that of unassisted clinicians
  (top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study
  arms, the DDx quality score was higher for clinicians assisted by our LLM
  (top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%)
  (McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p =
  0.03). Further, clinicians assisted by our LLM arrived at more comprehensive
  differential lists than those without its assistance. Our study suggests that
  our LLM for DDx has potential to improve clinicians' diagnostic reasoning and
  accuracy in challenging cases, meriting further real-world evaluation for its
  ability to empower physicians and widen patients' access to specialist-level
  expertise.
paper_ai_summary: An LLM optimized for diagnostic reasoning improves clinicians' differential diagnosis accuracy and comprehensiveness in challenging medical cases compared to standard tools.
paper_ai_keywords: LLM diagnostic reasoning differential diagnosis DDx top-10 accuracy McNemar's Test