Large Language Models in the Clinic: A Comprehensive Benchmark
<|reference_start|>Large Language Models in the Clinic: A Comprehensive Benchmark: The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering (QA) task with answer options for evaluation. However, many clinical decisions involve answering open-ended questions without pre-set options. To better understand LLMs in the clinic, we construct a benchmark ClinicBench. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. Furthermore, we construct six novel datasets and complex clinical tasks that are close to real-world practice, i.e., referral QA, treatment recommendation, hospitalization (long document) summarization, patient education, pharmacology QA and drug interaction for emerging drugs. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings. Finally, we invite medical experts to evaluate the clinical usefulness of LLMs.<|reference_end|>
@article{liu2024large,
    title={Large Language Models in the Clinic: A Comprehensive Benchmark},
    author={Fenglin Liu, Zheng Li, Hongjian Zhou, Qingyu Yin, Jingfeng Yang,
  Xianfeng Tang, Chen Luo, Ming Zeng, Haoming Jiang, Yifan Gao, Priyanka Nigam,
  Sreyashi Nag, Bing Yin, Yining Hua, Xuan Zhou, Omid Rohanian, Anshul Thakur,
  Lei Clifton, David A. Clifton},
    journal={arXiv preprint arXiv:2405.00716},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.00716},
    primaryClass={cs.CL cs.AI}
}