title: MedCaseReasoning: Evaluating and learning diagnostic reasoning from
    clinical case reports
summary: Doctors and patients alike increasingly use Large Language Models (LLMs) to
  diagnose clinical cases. However, unlike domains such as math or coding, where
  correctness can be objectively defined by the final answer, medical diagnosis
  requires both the outcome and the reasoning process to be accurate. Currently,
  widely used medical benchmarks like MedQA and MMLU assess only accuracy in the
  final answer, overlooking the quality and faithfulness of the clinical
  reasoning process. To address this limitation, we introduce MedCaseReasoning,
  the first open-access dataset for evaluating LLMs on their ability to align
  with clinician-authored diagnostic reasoning. The dataset includes 14,489
  diagnostic question-and-answer cases, each paired with detailed reasoning
  statements derived from open-access medical case reports. We evaluate
  state-of-the-art reasoning LLMs on MedCaseReasoning and find significant
  shortcomings in their diagnoses and reasoning: for instance, the top-performing
  open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy
  and mentions only 64% of the clinician reasoning statements (recall). However,
  we demonstrate that fine-tuning LLMs on the reasoning traces derived from
  MedCaseReasoning significantly improves diagnostic accuracy and clinical
  reasoning recall by an average relative gain of 29% and 41%, respectively. The
  open-source dataset, code, and models are available at
  https://github.com/kevinwu23/Stanford-MedCaseReasoning.
paper_authors._id: 682ce238d3fa4ecfd67b99d0
paper_authors.name: Kevin Wu
paper_authors._id: 682ce238d3fa4ecfd67b99d1
paper_authors.name: Eric Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:41:38.271Z
paper_authors._id: 61c4cf120cd370e91fc6f833
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61c4cf120cd370e91fc6f833/Q7tyj2Q0qK5-FmEDZ34WI.jpeg
paper_authors.fullname: Eric Wu
paper_authors.type: user
paper_authors.user: ericwu09
paper_authors._id: 682ce238d3fa4ecfd67b99d2
paper_authors.name: Rahul Thapa
paper_authors._id: 682ce238d3fa4ecfd67b99d3
paper_authors.name: Kevin Wei
paper_authors._id: 682ce238d3fa4ecfd67b99d4
paper_authors.name: Angela Zhang
paper_authors._id: 682ce238d3fa4ecfd67b99d5
paper_authors.name: Arvind Suresh
paper_authors._id: 682ce238d3fa4ecfd67b99d6
paper_authors.name: Jacqueline J. Tao
paper_authors._id: 682ce238d3fa4ecfd67b99d7
paper_authors.name: Min Woo Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T20:41:36.732Z
paper_authors._id: 65ac61120844d9e0d67a9f89
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ac61120844d9e0d67a9f89/OzA_yxFoeuyhmc3xdYYdP.jpeg
paper_authors.fullname: Min Woo Sun
paper_authors.type: user
paper_authors.user: minwoosun
paper_authors._id: 682ce238d3fa4ecfd67b99d8
paper_authors.name: Alejandro Lozano
paper_authors._id: 682ce238d3fa4ecfd67b99d9
paper_authors.name: James Zou
paper_title: MedCaseReasoning: Evaluating and learning diagnostic reasoning from
    clinical case reports
paper_summary: Doctors and patients alike increasingly use Large Language Models (LLMs) to
  diagnose clinical cases. However, unlike domains such as math or coding, where
  correctness can be objectively defined by the final answer, medical diagnosis
  requires both the outcome and the reasoning process to be accurate. Currently,
  widely used medical benchmarks like MedQA and MMLU assess only accuracy in the
  final answer, overlooking the quality and faithfulness of the clinical
  reasoning process. To address this limitation, we introduce MedCaseReasoning,
  the first open-access dataset for evaluating LLMs on their ability to align
  with clinician-authored diagnostic reasoning. The dataset includes 14,489
  diagnostic question-and-answer cases, each paired with detailed reasoning
  statements derived from open-access medical case reports. We evaluate
  state-of-the-art reasoning LLMs on MedCaseReasoning and find significant
  shortcomings in their diagnoses and reasoning: for instance, the top-performing
  open-source model, DeepSeek-R1, achieves only 48% 10-shot diagnostic accuracy
  and mentions only 64% of the clinician reasoning statements (recall). However,
  we demonstrate that fine-tuning LLMs on the reasoning traces derived from
  MedCaseReasoning significantly improves diagnostic accuracy and clinical
  reasoning recall by an average relative gain of 29% and 41%, respectively. The
  open-source dataset, code, and models are available at
  https://github.com/kevinwu23/Stanford-MedCaseReasoning.
paper_ai_summary: MedCaseReasoning is an open-access dataset that evaluates LLMs on diagnostic accuracy and clinical reasoning, revealing that fine-tuning on reasoning traces improves model performance.
paper_ai_keywords: Large Language Models (LLMs) medical benchmarks MedQA MMLU MedCaseReasoning diagnostic question-and-answer cases reasoning LLMs DeepSeek-R1 diagnostic accuracy recall fine-tuning reasoning traces