MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical
  Applications
The rapid development of Large Language Models (LLMs) for healthcare
applications has spurred calls for holistic evaluation beyond frequently-cited
benchmarks like USMLE, to better reflect real-world performance. While
real-world assessments are valuable indicators of utility, they often lag
behind the pace of LLM evolution, likely rendering findings obsolete upon
deployment. This temporal disconnect necessitates a comprehensive upfront
evaluation that can guide model selection for specific clinical applications.
We introduce MEDIC, a framework assessing LLMs across five critical dimensions
of clinical competence: medical reasoning, ethics and bias, data and language
understanding, in-context learning, and clinical safety. MEDIC features a novel
cross-examination framework quantifying LLM performance across areas like
coverage and hallucination detection, without requiring reference outputs. We
apply MEDIC to evaluate LLMs on medical question-answering, safety,
summarization, note generation, and other tasks. Our results show performance
disparities across model sizes, baseline vs medically finetuned models, and
have implications on model selection for applications requiring specific model
strengths, such as low hallucination or lower cost of inference. MEDIC's
multifaceted evaluation reveals these performance trade-offs, bridging the gap
between theoretical capabilities and practical implementation in healthcare
settings, ensuring that the most promising models are identified and adapted
for diverse healthcare applications.
66e263843b7e36b011d0f877
Praveen K Kanithi
claimed_verified
2024-09-12T07:12:02.324Z
65280984b794fe3d06544d77
https://cdn-avatars.huggingface.co/v1/production/uploads/65280984b794fe3d06544d77/tyrxbxtDG02On1uiRaVbL.jpeg
Praveenkumar
user
pkanithi
66e263843b7e36b011d0f878
Clément Christophe
admin_assigned
2024-09-12T07:13:15.223Z
628e39f4b1596566033b8d7b
https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg
Clément Christophe
user
cchristophe
66e263843b7e36b011d0f879
Marco AF Pimentel
admin_assigned
2024-09-12T07:17:25.568Z
6454faafa13edf669cd74f36
/avatars/355d16e28ca9cf5891368e43bcda6de5.svg
Marco Pimentel
user
mpimentel
66e263843b7e36b011d0f87a
Tathagata Raha
admin_assigned
2024-09-12T07:17:31.862Z
5f5f6c113c67af20d9945afb
/avatars/06b2eb3a5d27864280d4d02e6d00d782.svg
Tathagata Raha
user
tathagataraha
66e263843b7e36b011d0f87b
Nada Saadi
admin_assigned
2024-09-12T07:17:37.373Z
66bb35988b09ede0b7b92313
https://cdn-avatars.huggingface.co/v1/production/uploads/66bb35988b09ede0b7b92313/M06mQ3ifyRwuladTNwMS2.png
Nada Saadi
user
Nadas31
66e263843b7e36b011d0f87c
Hamza Javed
claimed_verified
2024-09-12T13:30:27.419Z
65e9aea6a00ce37e228c1a48
/avatars/af724095ec8d60e7ecbf3ae097419d93.svg
Hamza Javed
user
hjaved202
66e263843b7e36b011d0f87d
Svetlana Maslenkova
admin_assigned
2024-09-12T07:18:03.748Z
6506cfafd55dd4e15caeea09
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uTkC6G8Cj51av4i7kAaI8.png
Svetlana Maslenkova
user
maslenkovas
66e263843b7e36b011d0f87e
Nasir Hayat
admin_assigned
2024-09-12T07:18:15.965Z
647437a26a972f252de6b0ce
/avatars/02e6bed173eee14a18e30e0d247b8aa1.svg
Nasir Hayat
user
nasirhayat
66e263843b7e36b011d0f87f
Ronnie Rajan
admin_assigned
2024-09-12T07:18:22.360Z
65281d6ef61ca80b9c2ee707
/avatars/090ea7210a4bb6549b0f7fee71525625.svg
Ronnie Rajan
user
ronnierajan
66e263843b7e36b011d0f880
Shadab Khan
MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical
  Applications
The rapid development of Large Language Models (LLMs) for healthcare
applications has spurred calls for holistic evaluation beyond frequently-cited
benchmarks like USMLE, to better reflect real-world performance. While
real-world assessments are valuable indicators of utility, they often lag
behind the pace of LLM evolution, likely rendering findings obsolete upon
deployment. This temporal disconnect necessitates a comprehensive upfront
evaluation that can guide model selection for specific clinical applications.
We introduce MEDIC, a framework assessing LLMs across five critical dimensions
of clinical competence: medical reasoning, ethics and bias, data and language
understanding, in-context learning, and clinical safety. MEDIC features a novel
cross-examination framework quantifying LLM performance across areas like
coverage and hallucination detection, without requiring reference outputs. We
apply MEDIC to evaluate LLMs on medical question-answering, safety,
summarization, note generation, and other tasks. Our results show performance
disparities across model sizes, baseline vs medically finetuned models, and
have implications on model selection for applications requiring specific model
strengths, such as low hallucination or lower cost of inference. MEDIC's
multifaceted evaluation reveals these performance trade-offs, bridging the gap
between theoretical capabilities and practical implementation in healthcare
settings, ensuring that the most promising models are identified and adapted
for diverse healthcare applications.
MEDIC framework evaluates Large Language Models across five clinical dimensions to guide model selection in healthcare applications, identifying performance trade-offs and ensuring practical implementation.
Large Language Models MEDIC medical reasoning ethics and bias data and language understanding in-context learning clinical safety cross-examination framework hallucination detection medical question-answering safety summarization note generation