MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for
  Complex Medical Reasoning
Large Language Models (LLMs) have shown impressive performance on existing
medical question-answering benchmarks. This high performance makes it
increasingly difficult to meaningfully evaluate and differentiate advanced
methods. We present MedAgentsBench, a benchmark that focuses on challenging
medical questions requiring multi-step clinical reasoning, diagnosis
formulation, and treatment planning-scenarios where current models still
struggle despite their strong performance on standard tests. Drawing from seven
established medical datasets, our benchmark addresses three key limitations in
existing evaluations: (1) the prevalence of straightforward questions where
even base models achieve high performance, (2) inconsistent sampling and
evaluation protocols across studies, and (3) lack of systematic analysis of the
interplay between performance, cost, and inference time. Through experiments
with various base models and reasoning methods, we demonstrate that the latest
thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in
complex medical reasoning tasks. Additionally, advanced search-based agent
methods offer promising performance-to-cost ratios compared to traditional
approaches. Our analysis reveals substantial performance gaps between model
families on complex questions and identifies optimal model selections for
different computational constraints. Our benchmark and evaluation framework are
publicly available at https://github.com/gersteinlab/medagents-benchmark.
67cfd1934fed2b7e3e4cbb34
Xiangru Tang
extracted_pending
2025-03-11T06:00:52.457Z
63357c608adfa81faf2ac180
/avatars/ae0314c644f882251baf59b9134fd36f.svg
Xiangru Tang
user
RTT1
67cfd1934fed2b7e3e4cbb35
Daniel Shao
67cfd1934fed2b7e3e4cbb36
Jiwoong Sohn
claimed_verified
2025-06-16T12:56:37.417Z
64660e9ecf550af36eb2b774
https://cdn-avatars.huggingface.co/v1/production/uploads/64660e9ecf550af36eb2b774/tUySYpqg7FQQ_evLOWA70.jpeg
Jiwoong Sohn
user
jw-sohn
67cfd1934fed2b7e3e4cbb37
Jiapeng Chen
67cfd1934fed2b7e3e4cbb38
Jiayi Zhang
claimed_verified
2025-07-29T07:12:15.352Z
65f40e83653c231cbaf7defe
/avatars/afa5ce72324112739e539865c9aee26b.svg
Jiayi Zhang
user
didiforhugface
67cfd1934fed2b7e3e4cbb39
Jinyu Xiang
claimed_verified
2025-04-08T06:57:54.940Z
649ea7106282cb41e77760bc
https://cdn-avatars.huggingface.co/v1/production/uploads/649ea7106282cb41e77760bc/HlWjaqxr03ob93vdKg_LQ.jpeg
Isaac
user
XiangJinYu
67cfd1934fed2b7e3e4cbb3a
Fang Wu
claimed_verified
2025-07-23T08:38:21.142Z
675e0d5cdd3e9eeed6954f5a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7oMEoBmaFiCR9K2q9Z_7q.png
Fang Wu
user
fangwu97
67cfd1934fed2b7e3e4cbb3b
Yilun Zhao
claimed_verified
2025-03-31T08:15:57.888Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
67cfd1934fed2b7e3e4cbb3c
Chenglin Wu
claimed_verified
2025-04-08T06:57:57.846Z
64252594a4f3051f54dd4355
/avatars/c1f982b9b6be2956f6b6558be70d9ec3.svg
Wu
user
alexanderwu
67cfd1934fed2b7e3e4cbb3d
Wenqi Shi
claimed_verified
2025-03-11T08:21:16.321Z
65cae89119683f9817c049ea
https://cdn-avatars.huggingface.co/v1/production/uploads/65cae89119683f9817c049ea/A0XxjmaJldu28JhFvWmpP.jpeg
Wenqi Shi
user
wshi83
67cfd1934fed2b7e3e4cbb3e
Arman Cohan
67cfd1934fed2b7e3e4cbb3f
Mark Gerstein
MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for
  Complex Medical Reasoning
Large Language Models (LLMs) have shown impressive performance on existing
medical question-answering benchmarks. This high performance makes it
increasingly difficult to meaningfully evaluate and differentiate advanced
methods. We present MedAgentsBench, a benchmark that focuses on challenging
medical questions requiring multi-step clinical reasoning, diagnosis
formulation, and treatment planning-scenarios where current models still
struggle despite their strong performance on standard tests. Drawing from seven
established medical datasets, our benchmark addresses three key limitations in
existing evaluations: (1) the prevalence of straightforward questions where
even base models achieve high performance, (2) inconsistent sampling and
evaluation protocols across studies, and (3) lack of systematic analysis of the
interplay between performance, cost, and inference time. Through experiments
with various base models and reasoning methods, we demonstrate that the latest
thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in
complex medical reasoning tasks. Additionally, advanced search-based agent
methods offer promising performance-to-cost ratios compared to traditional
approaches. Our analysis reveals substantial performance gaps between model
families on complex questions and identifies optimal model selections for
different computational constraints. Our benchmark and evaluation framework are
publicly available at https://github.com/gersteinlab/medagents-benchmark.
MedAgentsBench evaluates advanced LLMs in medical reasoning through complex, multi-step clinical scenarios, demonstrating performance gaps and cost-effective search-based agent methods.
Large Language Models MedAgentsBench multi-step clinical reasoning diagnosis formulation treatment planning DeepSeek R1 OpenAI o3 search-based agent methods