Towards Accurate Differential Diagnosis with Large Language Models
An accurate differential diagnosis (DDx) is a cornerstone of medical care,
often reached through an iterative process of interpretation that combines
clinical history, physical examination, investigations and procedures.
Interactive interfaces powered by Large Language Models (LLMs) present new
opportunities to both assist and automate aspects of this process. In this
study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its
ability to generate a DDx alone or as an aid to clinicians. 20 clinicians
evaluated 302 challenging, real-world medical cases sourced from the New
England Journal of Medicine (NEJM) case reports. Each case report was read by
two clinicians, who were randomized to one of two assistive conditions: either
assistance from search engines and standard medical resources, or LLM
assistance in addition to these tools. All clinicians provided a baseline,
unassisted DDx prior to using the respective assistive tools. Our LLM for DDx
exhibited standalone performance that exceeded that of unassisted clinicians
(top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study
arms, the DDx quality score was higher for clinicians assisted by our LLM
(top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%)
(McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p =
0.03). Further, clinicians assisted by our LLM arrived at more comprehensive
differential lists than those without its assistance. Our study suggests that
our LLM for DDx has potential to improve clinicians' diagnostic reasoning and
accuracy in challenging cases, meriting further real-world evaluation for its
ability to empower physicians and widen patients' access to specialist-level
expertise.
656d32393dbac3a83d399ada
Daniel McDuff
656d32393dbac3a83d399adb
Mike Schaekermann
admin_assigned
2023-12-04T10:05:32.096Z
64ebb75f171e6c9862d261c9
/avatars/b7750f5d6281f906cfa5c717a44e2e13.svg
Mike Schaekermann
user
mikeschaekermann
656d32393dbac3a83d399adc
Tao Tu
656d32393dbac3a83d399add
Anil Palepu
656d32393dbac3a83d399ade
Amy Wang
656d32393dbac3a83d399adf
Jake Garrison
656d32393dbac3a83d399ae0
Karan Singhal
admin_assigned
2023-12-04T10:06:14.681Z
654bb16a48b474120258371c
/avatars/7ad4a33503533ddbb54c71b634fda89b.svg
Karan Singhal
user
karan1149
656d32393dbac3a83d399ae1
Yash Sharma
claimed_verified
2024-03-12T12:06:36.684Z
65a3178a534e60db99ec2cfc
/avatars/f0c5c364d8b2d1bdf7451973014f13be.svg
Yash Sharma
user
yashrma
656d32393dbac3a83d399ae2
Shekoofeh Azizi
656d32393dbac3a83d399ae3
Kavita Kulkarni
656d32393dbac3a83d399ae4
Le Hou
admin_assigned
2023-12-04T10:07:00.306Z
626b2402c804c432c1b5004c
/avatars/a119644422335e260af0ca4ef6399870.svg
Le Hou
user
solooobear
656d32393dbac3a83d399ae5
Yong Cheng
656d32393dbac3a83d399ae6
Yun Liu
656d32393dbac3a83d399ae7
S Sara Mahdavi
656d32393dbac3a83d399ae8
Sushant Prakash
656d32393dbac3a83d399ae9
Anupam Pathak
656d32393dbac3a83d399aea
Christopher Semturs
656d32393dbac3a83d399aeb
Shwetak Patel
656d32393dbac3a83d399aec
Dale R Webster
656d32393dbac3a83d399aed
Ewa Dominowska
656d32393dbac3a83d399aee
Juraj Gottweis
admin_assigned
2023-12-04T10:08:46.204Z
6381e100cae34727b9ca33ce
/avatars/f4e047e82e41ec507c6f51aef7560d87.svg
Juraj Gottweis
user
juro
656d32393dbac3a83d399aef
Joelle Barral
656d32393dbac3a83d399af0
Katherine Chou
656d32393dbac3a83d399af1
Greg S Corrado
656d32393dbac3a83d399af2
Yossi Matias
656d32393dbac3a83d399af3
Jake Sunshine
656d32393dbac3a83d399af4
Alan Karthikesalingam
656d32393dbac3a83d399af5
Vivek Natarajan
admin_assigned
2023-12-04T10:08:01.730Z
617a6af2941993035fbaf29c
/avatars/a389671494782f06ff8502c934090cc0.svg
Vivek Natarajan
user
vivnat
Towards Accurate Differential Diagnosis with Large Language Models
An accurate differential diagnosis (DDx) is a cornerstone of medical care,
often reached through an iterative process of interpretation that combines
clinical history, physical examination, investigations and procedures.
Interactive interfaces powered by Large Language Models (LLMs) present new
opportunities to both assist and automate aspects of this process. In this
study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its
ability to generate a DDx alone or as an aid to clinicians. 20 clinicians
evaluated 302 challenging, real-world medical cases sourced from the New
England Journal of Medicine (NEJM) case reports. Each case report was read by
two clinicians, who were randomized to one of two assistive conditions: either
assistance from search engines and standard medical resources, or LLM
assistance in addition to these tools. All clinicians provided a baseline,
unassisted DDx prior to using the respective assistive tools. Our LLM for DDx
exhibited standalone performance that exceeded that of unassisted clinicians
(top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study
arms, the DDx quality score was higher for clinicians assisted by our LLM
(top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%)
(McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p =
0.03). Further, clinicians assisted by our LLM arrived at more comprehensive
differential lists than those without its assistance. Our study suggests that
our LLM for DDx has potential to improve clinicians' diagnostic reasoning and
accuracy in challenging cases, meriting further real-world evaluation for its
ability to empower physicians and widen patients' access to specialist-level
expertise.
An LLM optimized for diagnostic reasoning improves clinicians' differential diagnosis accuracy and comprehensiveness in challenging medical cases compared to standard tools.
LLM diagnostic reasoning differential diagnosis DDx top-10 accuracy McNemar's Test