title: MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for
    Complex Medical Reasoning
summary: Large Language Models (LLMs) have shown impressive performance on existing
  medical question-answering benchmarks. This high performance makes it
  increasingly difficult to meaningfully evaluate and differentiate advanced
  methods. We present MedAgentsBench, a benchmark that focuses on challenging
  medical questions requiring multi-step clinical reasoning, diagnosis
  formulation, and treatment planning-scenarios where current models still
  struggle despite their strong performance on standard tests. Drawing from seven
  established medical datasets, our benchmark addresses three key limitations in
  existing evaluations: (1) the prevalence of straightforward questions where
  even base models achieve high performance, (2) inconsistent sampling and
  evaluation protocols across studies, and (3) lack of systematic analysis of the
  interplay between performance, cost, and inference time. Through experiments
  with various base models and reasoning methods, we demonstrate that the latest
  thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in
  complex medical reasoning tasks. Additionally, advanced search-based agent
  methods offer promising performance-to-cost ratios compared to traditional
  approaches. Our analysis reveals substantial performance gaps between model
  families on complex questions and identifies optimal model selections for
  different computational constraints. Our benchmark and evaluation framework are
  publicly available at https://github.com/gersteinlab/medagents-benchmark.
paper_authors._id: 67cfd1934fed2b7e3e4cbb34
paper_authors.name: Xiangru Tang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-11T06:00:52.457Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 67cfd1934fed2b7e3e4cbb35
paper_authors.name: Daniel Shao
paper_authors._id: 67cfd1934fed2b7e3e4cbb36
paper_authors.name: Jiwoong Sohn
paper_authors._id: 67cfd1934fed2b7e3e4cbb37
paper_authors.name: Jiapeng Chen
paper_authors._id: 67cfd1934fed2b7e3e4cbb38
paper_authors.name: Jiayi Zhang
paper_authors._id: 67cfd1934fed2b7e3e4cbb39
paper_authors.name: Jinyu Xiang
paper_authors._id: 67cfd1934fed2b7e3e4cbb3a
paper_authors.name: Fang Wu
paper_authors._id: 67cfd1934fed2b7e3e4cbb3b
paper_authors.name: Yilun Zhao
paper_authors._id: 67cfd1934fed2b7e3e4cbb3c
paper_authors.name: Chenglin Wu
paper_authors._id: 67cfd1934fed2b7e3e4cbb3d
paper_authors.name: Wenqi Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:21:16.321Z
paper_authors._id: 65cae89119683f9817c049ea
paper_authors.avatarUrl: /avatars/b08b10d7c72e2cf1108147e659411b32.svg
paper_authors.fullname: Wenqi Shi
paper_authors.type: user
paper_authors.user: wshi83
paper_authors._id: 67cfd1934fed2b7e3e4cbb3e
paper_authors.name: Arman Cohan
paper_authors._id: 67cfd1934fed2b7e3e4cbb3f
paper_authors.name: Mark Gerstein
paper_title: MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for
    Complex Medical Reasoning
paper_summary: Large Language Models (LLMs) have shown impressive performance on existing
  medical question-answering benchmarks. This high performance makes it
  increasingly difficult to meaningfully evaluate and differentiate advanced
  methods. We present MedAgentsBench, a benchmark that focuses on challenging
  medical questions requiring multi-step clinical reasoning, diagnosis
  formulation, and treatment planning-scenarios where current models still
  struggle despite their strong performance on standard tests. Drawing from seven
  established medical datasets, our benchmark addresses three key limitations in
  existing evaluations: (1) the prevalence of straightforward questions where
  even base models achieve high performance, (2) inconsistent sampling and
  evaluation protocols across studies, and (3) lack of systematic analysis of the
  interplay between performance, cost, and inference time. Through experiments
  with various base models and reasoning methods, we demonstrate that the latest
  thinking models, DeepSeek R1 and OpenAI o3, exhibit exceptional performance in
  complex medical reasoning tasks. Additionally, advanced search-based agent
  methods offer promising performance-to-cost ratios compared to traditional
  approaches. Our analysis reveals substantial performance gaps between model
  families on complex questions and identifies optimal model selections for
  different computational constraints. Our benchmark and evaluation framework are
  publicly available at https://github.com/gersteinlab/medagents-benchmark.
paper_ai_keywords: MedAgentsBench multi-step clinical reasoning diagnosis formulation treatment planning MedAgentsBench DeepSeek R1 OpenAI o3 advanced search-based agent methods