title: MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical
    Applications
summary: The rapid development of Large Language Models (LLMs) for healthcare
  applications has spurred calls for holistic evaluation beyond frequently-cited
  benchmarks like USMLE, to better reflect real-world performance. While
  real-world assessments are valuable indicators of utility, they often lag
  behind the pace of LLM evolution, likely rendering findings obsolete upon
  deployment. This temporal disconnect necessitates a comprehensive upfront
  evaluation that can guide model selection for specific clinical applications.
  We introduce MEDIC, a framework assessing LLMs across five critical dimensions
  of clinical competence: medical reasoning, ethics and bias, data and language
  understanding, in-context learning, and clinical safety. MEDIC features a novel
  cross-examination framework quantifying LLM performance across areas like
  coverage and hallucination detection, without requiring reference outputs. We
  apply MEDIC to evaluate LLMs on medical question-answering, safety,
  summarization, note generation, and other tasks. Our results show performance
  disparities across model sizes, baseline vs medically finetuned models, and
  have implications on model selection for applications requiring specific model
  strengths, such as low hallucination or lower cost of inference. MEDIC's
  multifaceted evaluation reveals these performance trade-offs, bridging the gap
  between theoretical capabilities and practical implementation in healthcare
  settings, ensuring that the most promising models are identified and adapted
  for diverse healthcare applications.
paper_authors._id: 66e263843b7e36b011d0f877
paper_authors.name: Praveen K Kanithi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-12T07:12:02.324Z
paper_authors._id: 65280984b794fe3d06544d77
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65280984b794fe3d06544d77/tyrxbxtDG02On1uiRaVbL.jpeg
paper_authors.fullname: Praveenkumar
paper_authors.type: user
paper_authors.user: pkanithi
paper_authors._id: 66e263843b7e36b011d0f878
paper_authors.name: Clément Christophe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:13:15.223Z
paper_authors._id: 628e39f4b1596566033b8d7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e39f4b1596566033b8d7b/-Y807up1cgMmAQsczdOPn.jpeg
paper_authors.fullname: Clément Christophe
paper_authors.type: user
paper_authors.user: cchristophe
paper_authors._id: 66e263843b7e36b011d0f879
paper_authors.name: Marco AF Pimentel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:17:25.568Z
paper_authors._id: 6454faafa13edf669cd74f36
paper_authors.avatarUrl: /avatars/355d16e28ca9cf5891368e43bcda6de5.svg
paper_authors.fullname: Marco Pimentel
paper_authors.type: user
paper_authors.user: mpimentel
paper_authors._id: 66e263843b7e36b011d0f87a
paper_authors.name: Tathagata Raha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:17:31.862Z
paper_authors._id: 5f5f6c113c67af20d9945afb
paper_authors.avatarUrl: /avatars/06b2eb3a5d27864280d4d02e6d00d782.svg
paper_authors.fullname: Tathagata Raha
paper_authors.type: user
paper_authors.user: tathagataraha
paper_authors._id: 66e263843b7e36b011d0f87b
paper_authors.name: Nada Saadi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:17:37.373Z
paper_authors._id: 66bb35988b09ede0b7b92313
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66bb35988b09ede0b7b92313/M06mQ3ifyRwuladTNwMS2.png
paper_authors.fullname: Nada Saadi
paper_authors.type: user
paper_authors.user: Nadas31
paper_authors._id: 66e263843b7e36b011d0f87c
paper_authors.name: Hamza Javed
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-12T13:30:27.419Z
paper_authors._id: 65e9aea6a00ce37e228c1a48
paper_authors.avatarUrl: /avatars/af724095ec8d60e7ecbf3ae097419d93.svg
paper_authors.fullname: Hamza Javed
paper_authors.type: user
paper_authors.user: hjaved202
paper_authors._id: 66e263843b7e36b011d0f87d
paper_authors.name: Svetlana Maslenkova
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:18:03.748Z
paper_authors._id: 6506cfafd55dd4e15caeea09
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/uTkC6G8Cj51av4i7kAaI8.png
paper_authors.fullname: Svetlana Maslenkova
paper_authors.type: user
paper_authors.user: maslenkovas
paper_authors._id: 66e263843b7e36b011d0f87e
paper_authors.name: Nasir Hayat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:18:15.965Z
paper_authors._id: 647437a26a972f252de6b0ce
paper_authors.avatarUrl: /avatars/02e6bed173eee14a18e30e0d247b8aa1.svg
paper_authors.fullname: Nasir Hayat
paper_authors.type: user
paper_authors.user: nasirhayat
paper_authors._id: 66e263843b7e36b011d0f87f
paper_authors.name: Ronnie Rajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-12T07:18:22.360Z
paper_authors._id: 65281d6ef61ca80b9c2ee707
paper_authors.avatarUrl: /avatars/090ea7210a4bb6549b0f7fee71525625.svg
paper_authors.fullname: Ronnie Rajan
paper_authors.type: user
paper_authors.user: ronnierajan
paper_authors._id: 66e263843b7e36b011d0f880
paper_authors.name: Shadab Khan
paper_title: MEDIC: Towards a Comprehensive Framework for Evaluating LLMs in Clinical
    Applications
paper_summary: The rapid development of Large Language Models (LLMs) for healthcare
  applications has spurred calls for holistic evaluation beyond frequently-cited
  benchmarks like USMLE, to better reflect real-world performance. While
  real-world assessments are valuable indicators of utility, they often lag
  behind the pace of LLM evolution, likely rendering findings obsolete upon
  deployment. This temporal disconnect necessitates a comprehensive upfront
  evaluation that can guide model selection for specific clinical applications.
  We introduce MEDIC, a framework assessing LLMs across five critical dimensions
  of clinical competence: medical reasoning, ethics and bias, data and language
  understanding, in-context learning, and clinical safety. MEDIC features a novel
  cross-examination framework quantifying LLM performance across areas like
  coverage and hallucination detection, without requiring reference outputs. We
  apply MEDIC to evaluate LLMs on medical question-answering, safety,
  summarization, note generation, and other tasks. Our results show performance
  disparities across model sizes, baseline vs medically finetuned models, and
  have implications on model selection for applications requiring specific model
  strengths, such as low hallucination or lower cost of inference. MEDIC's
  multifaceted evaluation reveals these performance trade-offs, bridging the gap
  between theoretical capabilities and practical implementation in healthcare
  settings, ensuring that the most promising models are identified and adapted
  for diverse healthcare applications.
paper_ai_keywords: Large Language Models (LLMs) MEDIC medical reasoning ethics and bias data and language understanding in-context learning clinical safety cross-examination framework hallucination detection medical question-answering note generation