title: ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical
    Prediction?
summary: Large Language Models (LLMs) hold great promise to revolutionize current
  clinical systems for their superior capacities on medical text processing tasks
  and medical licensing exams. Meanwhile, traditional ML models such as SVM and
  XGBoost have still been mainly adopted in clinical prediction tasks. An
  emerging question is Can LLMs beat traditional ML models in clinical
  prediction? Thus, we build a new benchmark ClinicalBench to comprehensively
  study the clinical predictive modeling capacities of both general-purpose and
  medical LLMs, and compare them with traditional ML models. ClinicalBench
  embraces three common clinical prediction tasks, two databases, 14
  general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through
  extensive empirical investigation, we discover that both general-purpose and
  medical LLMs, even with different model scales, diverse prompting or
  fine-tuning strategies, still cannot beat traditional ML models in clinical
  prediction yet, shedding light on their potential deficiency in clinical
  reasoning and decision-making. We call for caution when practitioners adopt
  LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap
  between LLMs' development for healthcare and real-world clinical practice.
paper_authors._id: 6736bdee81fd1225f7c1dc6d
paper_authors.name: Canyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:21:19.591Z
paper_authors._id: 6483af58571c2dcfa98cae82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483af58571c2dcfa98cae82/d8Vnh_EG1KxeEoNAAQXHZ.jpeg
paper_authors.fullname: Canyu Chen
paper_authors.type: user
paper_authors.user: canyuchen
paper_authors._id: 6736bdee81fd1225f7c1dc6e
paper_authors.name: Jian Yu
paper_authors._id: 6736bdee81fd1225f7c1dc6f
paper_authors.name: Shan Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T17:25:44.224Z
paper_authors._id: 63600b93d9e4214b9a67807c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63600b93d9e4214b9a67807c/BovBE1oiZPdU8Td9I2YIn.jpeg
paper_authors.fullname: Shan Chen
paper_authors.type: user
paper_authors.user: shanchen
paper_authors._id: 6736bdee81fd1225f7c1dc70
paper_authors.name: Che Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:17:57.848Z
paper_authors._id: 631b9ff5824f2502e3557c7e
paper_authors.avatarUrl: /avatars/076043c9dba07644a570692563ef8114.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: che111
paper_authors._id: 6736bdee81fd1225f7c1dc71
paper_authors.name: Zhongwei Wan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:04:59.950Z
paper_authors._id: 64b20d0a372d4340773ff67e
paper_authors.avatarUrl: /avatars/0f194f0f3345e8eb3f14fd99a1fb49f8.svg
paper_authors.fullname: Zhongwei Wang
paper_authors.type: user
paper_authors.user: hell-driver
paper_authors._id: 6736bdee81fd1225f7c1dc72
paper_authors.name: Danielle Bitterman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:01:30.067Z
paper_authors._id: 6570cd33f3853f99bc305e55
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6570cd33f3853f99bc305e55/70ZnaOtdtmy8DeWVJpCKb.png
paper_authors.fullname: Danielle Bitterman
paper_authors.type: user
paper_authors.user: daniellebitt
paper_authors._id: 6736bdee81fd1225f7c1dc73
paper_authors.name: Fei Wang
paper_authors._id: 6736bdee81fd1225f7c1dc74
paper_authors.name: Kai Shu
paper_title: ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical
    Prediction?
paper_summary: Large Language Models (LLMs) hold great promise to revolutionize current
  clinical systems for their superior capacities on medical text processing tasks
  and medical licensing exams. Meanwhile, traditional ML models such as SVM and
  XGBoost have still been mainly adopted in clinical prediction tasks. An
  emerging question is Can LLMs beat traditional ML models in clinical
  prediction? Thus, we build a new benchmark ClinicalBench to comprehensively
  study the clinical predictive modeling capacities of both general-purpose and
  medical LLMs, and compare them with traditional ML models. ClinicalBench
  embraces three common clinical prediction tasks, two databases, 14
  general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through
  extensive empirical investigation, we discover that both general-purpose and
  medical LLMs, even with different model scales, diverse prompting or
  fine-tuning strategies, still cannot beat traditional ML models in clinical
  prediction yet, shedding light on their potential deficiency in clinical
  reasoning and decision-making. We call for caution when practitioners adopt
  LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap
  between LLMs' development for healthcare and real-world clinical practice.