A Dataset and Benchmark for Hospital Course Summarization with Adapted Large Language Models
<|reference_start|>A Dataset and Benchmark for Hospital Course Summarization with Adapted Large Language Models: Brief hospital course (BHC) summaries are clinical documents that summarize a patient's hospital stay. While large language models (LLMs) depict remarkable capabilities in automating real-world tasks, their capabilities for healthcare applications such as synthesizing BHCs from clinical notes have not been shown. We introduce a novel pre-processed dataset, the MIMIC-IV-BHC, encapsulating clinical note and brief hospital course (BHC) pairs to adapt LLMs for BHC synthesis. Furthermore, we introduce a benchmark of the summarization performance of two general-purpose LLMs and three healthcare-adapted LLMs.   Using clinical notes as input, we apply prompting-based (using in-context learning) and fine-tuning-based adaptation strategies to three open-source LLMs (Clinical-T5-Large, Llama2-13B, FLAN-UL2) and two proprietary LLMs (GPT-3.5, GPT-4). We evaluate these LLMs across multiple context-length inputs using natural language similarity metrics. We further conduct a clinical study with five clinicians, comparing clinician-written and LLM-generated BHCs across 30 samples, focusing on their potential to enhance clinical decision-making through improved summary quality. We observe that the Llama2-13B fine-tuned LLM outperforms other domain-adapted models given quantitative evaluation metrics of BLEU and BERT-Score. GPT-4 with in-context learning shows more robustness to increasing context lengths of clinical note inputs than fine-tuned Llama2-13B. Despite comparable quantitative metrics, the reader study depicts a significant preference for summaries generated by GPT-4 with in-context learning compared to both Llama2-13B fine-tuned summaries and the original summaries, highlighting the need for qualitative clinical evaluation.<|reference_end|>
@article{aali2024a,
    title={A Dataset and Benchmark for Hospital Course Summarization with Adapted
  Large Language Models},
    author={Asad Aali, Dave Van Veen, Yamin Ishraq Arefeen, Jason Hom, Christian
  Bluethgen, Eduardo Pontes Reis, Sergios Gatidis, Namuun Clifford, Joseph
  Daws, Arash S. Tehrani, Jangwon Kim, Akshay S. Chaudhari},
    journal={arXiv preprint arXiv:2403.05720},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.05720},
    primaryClass={cs.CL cs.AI cs.LG}
}