MedAlign: A Clinician-Generated Dataset for Instruction Following with
  Electronic Medical Records
The ability of large language models (LLMs) to follow natural language
instructions with human-level fluency suggests many opportunities in healthcare
to reduce administrative burden and improve quality of care. However,
evaluating LLMs on realistic text generation tasks for healthcare remains
challenging. Existing question answering datasets for electronic health record
(EHR) data fail to capture the complexity of information needs and
documentation burdens experienced by clinicians. To address these challenges,
we introduce MedAlign, a benchmark dataset of 983 natural language instructions
for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes
clinician-written reference responses for 303 instructions, and provides 276
longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to
evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality
of each LLM response. We found high error rates, ranging from 35% (GPT-4) to
68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k
context lengths for GPT-4. Finally, we report correlations between clinician
rankings and automated natural language generation metrics as a way to rank
LLMs without human review. We make MedAlign available under a research data use
agreement to enable LLM evaluations on tasks aligned with clinician needs and
preferences.
64eddd360a4f28000c596a09
Scott L. Fleming
admin_assigned
2023-08-29T12:42:54.737Z
64bff94fb878d31de52fc2ea
/avatars/a6b0b227e3c6e3a0e3aaf4f1f8a1d90f.svg
Scott Fleming
user
scottyf
64eddd360a4f28000c596a0a
Alejandro Lozano
admin_assigned
2023-08-29T12:43:36.376Z
643c3fb526f177a3e4181e05
/avatars/4f08ca15c5fbd2f8bb3d5c24f3fcc7b8.svg
Alejandro Lozano
user
lozanoe
64eddd360a4f28000c596a0b
William J. Haberkorn
64eddd360a4f28000c596a0c
Jenelle A. Jindal
64eddd360a4f28000c596a0d
Eduardo P. Reis
64eddd360a4f28000c596a0e
Rahul Thapa
64eddd360a4f28000c596a0f
Louis Blankemeier
admin_assigned
2023-08-29T13:20:41.112Z
6377044ea3b787faca443cc0
/avatars/5cd6cc70ad597dd83776018a452947d0.svg
Louis Blankemeier
user
louisblankemeier
64eddd360a4f28000c596a10
Julian Z. Genkins
64eddd360a4f28000c596a11
Ethan Steinberg
64eddd360a4f28000c596a12
Ashwin Nayak
64eddd360a4f28000c596a13
Birju S. Patel
64eddd360a4f28000c596a14
Chia-Chun Chiang
64eddd360a4f28000c596a15
Alison Callahan
claimed_verified
2023-08-29T17:36:19.743Z
64ee23743c4b9517ae80ec7a
/avatars/498d3f3c49b40559248460e6677075a2.svg
Alison Callahan
user
clssfr
64eddd360a4f28000c596a16
Zepeng Huo
admin_assigned
2023-08-29T13:38:48.743Z
63d1ca9e2036e44c44fe47a2
/avatars/78f1c4b7571a1d47d3d53982af2613dd.svg
Zepeng Huo
user
frazierhuo
64eddd360a4f28000c596a17
Sergios Gatidis
64eddd360a4f28000c596a18
Scott J. Adams
64eddd360a4f28000c596a19
Oluseyi Fayanju
64eddd360a4f28000c596a1a
Shreya J. Shah
64eddd360a4f28000c596a1b
Thomas Savage
64eddd360a4f28000c596a1c
Ethan Goh
admin_assigned
2023-08-29T13:27:18.117Z
647ed23eb0e96764589b8e08
/avatars/7a3b9c93384a7028151c1f0aa99181e2.svg
ethan goh
user
ethangoh
64eddd360a4f28000c596a1d
Akshay S. Chaudhari
admin_assigned
2023-08-29T13:26:52.370Z
6236533b76c8a780323af640
/avatars/18078ad26aa44312a4927160216e5943.svg
Akshay Chaudhari
user
akshaysc
64eddd360a4f28000c596a1e
Nima Aghaeepour
64eddd360a4f28000c596a1f
Christopher Sharp
64eddd360a4f28000c596a20
Michael A. Pfeffer
64eddd360a4f28000c596a21
Percy Liang
admin_assigned
2023-08-29T13:40:40.643Z
6409651b9e9f790c905b2335
/avatars/1fb8c80b60f21f65a0a027319101f236.svg
Percy Liang
user
percyliang
64eddd360a4f28000c596a22
Jonathan H. Chen
64eddd360a4f28000c596a23
Keith E. Morse
64eddd360a4f28000c596a24
Emma P. Brunskill
64eddd360a4f28000c596a25
Jason A. Fries
claimed_verified
2023-08-29T17:36:16.067Z
60836210e2b7cc3a117b0da0
https://cdn-avatars.huggingface.co/v1/production/uploads/1643137342884-60836210e2b7cc3a117b0da0.jpeg
Jason Alan Fries
user
jfries
64eddd360a4f28000c596a26
Nigam H. Shah
claimed_verified
2023-08-29T15:01:13.325Z
64ee047b22d9b8cec9fcd3cc
/avatars/dce9d03d9071017c8d0d53c670e5aed2.svg
Nigam Shah
user
drnigam
MedAlign: A Clinician-Generated Dataset for Instruction Following with
  Electronic Medical Records
The ability of large language models (LLMs) to follow natural language
instructions with human-level fluency suggests many opportunities in healthcare
to reduce administrative burden and improve quality of care. However,
evaluating LLMs on realistic text generation tasks for healthcare remains
challenging. Existing question answering datasets for electronic health record
(EHR) data fail to capture the complexity of information needs and
documentation burdens experienced by clinicians. To address these challenges,
we introduce MedAlign, a benchmark dataset of 983 natural language instructions
for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes
clinician-written reference responses for 303 instructions, and provides 276
longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to
evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality
of each LLM response. We found high error rates, ranging from 35% (GPT-4) to
68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k
context lengths for GPT-4. Finally, we report correlations between clinician
rankings and automated natural language generation metrics as a way to rank
LLMs without human review. We make MedAlign available under a research data use
agreement to enable LLM evaluations on tasks aligned with clinician needs and
preferences.
MedAlign, a curated dataset of natural language instructions for EHR data, evaluates the accuracy and quality of general domain LLMs in healthcare, revealing high error rates and context length dependencies.
large language models LLMs electronic health record EHR natural language instructions benchmark dataset clinician-written reference responses longitudinal EHRs instruction-response pairs context lengths automated natural language generation metrics