O1 Replication Journey -- Part 3: Inference-time Scaling for Medical
  Reasoning
Building upon our previous investigations of O1 replication (Part 1: Journey
Learning [Qin et al., 2024] and Part 2: Distillation [Huang et al., 2024]),
this work explores the potential of inference-time scaling in large language
models (LLMs) for medical reasoning tasks, ranging from diagnostic
decision-making to treatment planning. Through extensive experiments on medical
benchmarks of varying complexity (MedQA, Medbullets, and JAMA Clinical
Challenges), our investigation reveals several key insights: (1) Increasing
inference time does lead to improved performance. With a modest training set of
500 samples, our model yields substantial performance improvements of 6%-11%.
(2) Task complexity directly correlates with the required length of reasoning
chains, confirming the necessity of extended thought processes for challenging
problems. (3) The differential diagnoses generated by our model adhere to the
principles of the hypothetico-deductive method, producing a list of potential
conditions that may explain a patient's symptoms and systematically narrowing
these possibilities by evaluating the evidence. These findings demonstrate the
promising synergy between inference-time scaling and journey learning in
advancing LLMs' real-world clinical reasoning capabilities.
6785e65a7d58f2945af8034d
Zhongzhen Huang
admin_assigned
2025-01-14T09:10:03.906Z
620a0a2e8d5e5dfed284bc41
/avatars/ca2ddad577227103cdf263a4b7817b22.svg
huangzhongzhen
user
zongzi
6785e65a7d58f2945af8034e
Gui Geng
6785e65a7d58f2945af8034f
Shengyi Hua
6785e65a7d58f2945af80350
Zhen Huang
admin_assigned
2025-01-14T09:12:15.207Z
643581a4f3b08e267d990499
https://cdn-avatars.huggingface.co/v1/production/uploads/643581a4f3b08e267d990499/KRhB-48W4IPuB0bX16Ahj.png
Zhen Huang
user
ZhenHuang
6785e65a7d58f2945af80351
Haoyang Zou
admin_assigned
2025-01-14T09:12:22.954Z
65011b18af1e4021439af578
/avatars/ae1632c800ceeb43aa223e432f70bc6d.svg
Haoyang Zou
user
alanyoung058
6785e65a7d58f2945af80352
Shaoting Zhang
6785e65a7d58f2945af80353
Pengfei Liu
admin_assigned
2025-01-14T09:12:46.582Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
6785e65a7d58f2945af80354
Xiaofan Zhang
O1 Replication Journey -- Part 3: Inference-time Scaling for Medical
  Reasoning
Building upon our previous investigations of O1 replication (Part 1: Journey
Learning [Qin et al., 2024] and Part 2: Distillation [Huang et al., 2024]),
this work explores the potential of inference-time scaling in large language
models (LLMs) for medical reasoning tasks, ranging from diagnostic
decision-making to treatment planning. Through extensive experiments on medical
benchmarks of varying complexity (MedQA, Medbullets, and JAMA Clinical
Challenges), our investigation reveals several key insights: (1) Increasing
inference time does lead to improved performance. With a modest training set of
500 samples, our model yields substantial performance improvements of 6%-11%.
(2) Task complexity directly correlates with the required length of reasoning
chains, confirming the necessity of extended thought processes for challenging
problems. (3) The differential diagnoses generated by our model adhere to the
principles of the hypothetico-deductive method, producing a list of potential
conditions that may explain a patient's symptoms and systematically narrowing
these possibilities by evaluating the evidence. These findings demonstrate the
promising synergy between inference-time scaling and journey learning in
advancing LLMs' real-world clinical reasoning capabilities.
Inference-time scaling in large language models enhances medical reasoning performance by increasing the length of reasoning chains and adhering to the hypothetico-deductive method in generating differential diagnoses.
inference-time scaling large language models medical reasoning diagnostic decision-making treatment planning MedQA Medbullets JAMA Clinical Challenges reasoning chains hypothetico-deductive method differential diagnoses