title: MedAlign: A Clinician-Generated Dataset for Instruction Following with
    Electronic Medical Records
summary: The ability of large language models (LLMs) to follow natural language
  instructions with human-level fluency suggests many opportunities in healthcare
  to reduce administrative burden and improve quality of care. However,
  evaluating LLMs on realistic text generation tasks for healthcare remains
  challenging. Existing question answering datasets for electronic health record
  (EHR) data fail to capture the complexity of information needs and
  documentation burdens experienced by clinicians. To address these challenges,
  we introduce MedAlign, a benchmark dataset of 983 natural language instructions
  for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes
  clinician-written reference responses for 303 instructions, and provides 276
  longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to
  evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality
  of each LLM response. We found high error rates, ranging from 35% (GPT-4) to
  68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k
  context lengths for GPT-4. Finally, we report correlations between clinician
  rankings and automated natural language generation metrics as a way to rank
  LLMs without human review. We make MedAlign available under a research data use
  agreement to enable LLM evaluations on tasks aligned with clinician needs and
  preferences.
paper_authors._id: 64eddd360a4f28000c596a09
paper_authors.name: Scott L. Fleming
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-29T12:42:54.737Z
paper_authors._id: 64bff94fb878d31de52fc2ea
paper_authors.avatarUrl: /avatars/a6b0b227e3c6e3a0e3aaf4f1f8a1d90f.svg
paper_authors.fullname: Scott Fleming
paper_authors.type: user
paper_authors.user: scottyf
paper_authors._id: 64eddd360a4f28000c596a0a
paper_authors.name: Alejandro Lozano
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-29T12:43:36.376Z
paper_authors._id: 643c3fb526f177a3e4181e05
paper_authors.avatarUrl: /avatars/4f08ca15c5fbd2f8bb3d5c24f3fcc7b8.svg
paper_authors.fullname: Alejandro Lozano
paper_authors.type: user
paper_authors.user: lozanoe
paper_authors._id: 64eddd360a4f28000c596a0b
paper_authors.name: William J. Haberkorn
paper_authors._id: 64eddd360a4f28000c596a0c
paper_authors.name: Jenelle A. Jindal
paper_authors._id: 64eddd360a4f28000c596a0d
paper_authors.name: Eduardo P. Reis
paper_authors._id: 64eddd360a4f28000c596a0e
paper_authors.name: Rahul Thapa
paper_authors._id: 64eddd360a4f28000c596a0f
paper_authors.name: Louis Blankemeier
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-29T13:20:41.112Z
paper_authors._id: 6377044ea3b787faca443cc0
paper_authors.avatarUrl: /avatars/5cd6cc70ad597dd83776018a452947d0.svg
paper_authors.fullname: Louis Blankemeier
paper_authors.type: user
paper_authors.user: louisblankemeier
paper_authors._id: 64eddd360a4f28000c596a10
paper_authors.name: Julian Z. Genkins
paper_authors._id: 64eddd360a4f28000c596a11
paper_authors.name: Ethan Steinberg
paper_authors._id: 64eddd360a4f28000c596a12
paper_authors.name: Ashwin Nayak
paper_authors._id: 64eddd360a4f28000c596a13
paper_authors.name: Birju S. Patel
paper_authors._id: 64eddd360a4f28000c596a14
paper_authors.name: Chia-Chun Chiang
paper_authors._id: 64eddd360a4f28000c596a15
paper_authors.name: Alison Callahan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-29T17:36:19.743Z
paper_authors._id: 64ee23743c4b9517ae80ec7a
paper_authors.avatarUrl: /avatars/498d3f3c49b40559248460e6677075a2.svg
paper_authors.fullname: Alison Callahan
paper_authors.type: user
paper_authors.user: clssfr
paper_authors._id: 64eddd360a4f28000c596a16
paper_authors.name: Zepeng Huo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-29T13:38:48.743Z
paper_authors._id: 63d1ca9e2036e44c44fe47a2
paper_authors.avatarUrl: /avatars/78f1c4b7571a1d47d3d53982af2613dd.svg
paper_authors.fullname: Zepeng Huo
paper_authors.type: user
paper_authors.user: frazierhuo
paper_authors._id: 64eddd360a4f28000c596a17
paper_authors.name: Sergios Gatidis
paper_authors._id: 64eddd360a4f28000c596a18
paper_authors.name: Scott J. Adams
paper_authors._id: 64eddd360a4f28000c596a19
paper_authors.name: Oluseyi Fayanju
paper_authors._id: 64eddd360a4f28000c596a1a
paper_authors.name: Shreya J. Shah
paper_authors._id: 64eddd360a4f28000c596a1b
paper_authors.name: Thomas Savage
paper_authors._id: 64eddd360a4f28000c596a1c
paper_authors.name: Ethan Goh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-29T13:27:18.117Z
paper_authors._id: 647ed23eb0e96764589b8e08
paper_authors.avatarUrl: /avatars/7a3b9c93384a7028151c1f0aa99181e2.svg
paper_authors.fullname: ethan goh
paper_authors.type: user
paper_authors.user: ethangoh
paper_authors._id: 64eddd360a4f28000c596a1d
paper_authors.name: Akshay S. Chaudhari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-29T13:26:52.370Z
paper_authors._id: 6236533b76c8a780323af640
paper_authors.avatarUrl: /avatars/18078ad26aa44312a4927160216e5943.svg
paper_authors.fullname: Akshay Chaudhari
paper_authors.type: user
paper_authors.user: akshaysc
paper_authors._id: 64eddd360a4f28000c596a1e
paper_authors.name: Nima Aghaeepour
paper_authors._id: 64eddd360a4f28000c596a1f
paper_authors.name: Christopher Sharp
paper_authors._id: 64eddd360a4f28000c596a20
paper_authors.name: Michael A. Pfeffer
paper_authors._id: 64eddd360a4f28000c596a21
paper_authors.name: Percy Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-29T13:40:40.643Z
paper_authors._id: 6409651b9e9f790c905b2335
paper_authors.avatarUrl: /avatars/1fb8c80b60f21f65a0a027319101f236.svg
paper_authors.fullname: Percy Liang
paper_authors.type: user
paper_authors.user: percyliang
paper_authors._id: 64eddd360a4f28000c596a22
paper_authors.name: Jonathan H. Chen
paper_authors._id: 64eddd360a4f28000c596a23
paper_authors.name: Keith E. Morse
paper_authors._id: 64eddd360a4f28000c596a24
paper_authors.name: Emma P. Brunskill
paper_authors._id: 64eddd360a4f28000c596a25
paper_authors.name: Jason A. Fries
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-29T17:36:16.067Z
paper_authors._id: 60836210e2b7cc3a117b0da0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643137342884-60836210e2b7cc3a117b0da0.jpeg
paper_authors.fullname: Jason Alan Fries
paper_authors.type: user
paper_authors.user: jfries
paper_authors._id: 64eddd360a4f28000c596a26
paper_authors.name: Nigam H. Shah
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-29T15:01:13.325Z
paper_authors._id: 64ee047b22d9b8cec9fcd3cc
paper_authors.avatarUrl: /avatars/dce9d03d9071017c8d0d53c670e5aed2.svg
paper_authors.fullname: Nigam Shah
paper_authors.type: user
paper_authors.user: drnigam
paper_title: MedAlign: A Clinician-Generated Dataset for Instruction Following with
    Electronic Medical Records
paper_summary: The ability of large language models (LLMs) to follow natural language
  instructions with human-level fluency suggests many opportunities in healthcare
  to reduce administrative burden and improve quality of care. However,
  evaluating LLMs on realistic text generation tasks for healthcare remains
  challenging. Existing question answering datasets for electronic health record
  (EHR) data fail to capture the complexity of information needs and
  documentation burdens experienced by clinicians. To address these challenges,
  we introduce MedAlign, a benchmark dataset of 983 natural language instructions
  for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes
  clinician-written reference responses for 303 instructions, and provides 276
  longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to
  evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality
  of each LLM response. We found high error rates, ranging from 35% (GPT-4) to
  68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k
  context lengths for GPT-4. Finally, we report correlations between clinician
  rankings and automated natural language generation metrics as a way to rank
  LLMs without human review. We make MedAlign available under a research data use
  agreement to enable LLM evaluations on tasks aligned with clinician needs and
  preferences.
paper_ai_keywords: large language models (LLMs) natural language instructions human-level fluency healthcare administrative burden quality of care evaluating LLMs text generation tasks electronic health record (EHR) data comprehension complexity documentation burdens clinicians MedAlign benchmark dataset natural language instructions EHR data clinician-written reference responses longitudinal EHRs instruction-response pairs general domain LLMs accuracy quality of each LLM response error rates context lengths correlations between clinician rankings automated natural language generation metrics research data use agreement