title: A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?
summary: Large language models (LLMs) have exhibited remarkable capabilities across
  various domains and tasks, pushing the boundaries of our knowledge in learning
  and cognition. The latest model, OpenAI's o1, stands out as the first LLM with
  an internalized chain-of-thought technique using reinforcement learning
  strategies. While it has demonstrated surprisingly strong capabilities on
  various general language tasks, its performance in specialized fields such as
  medicine remains unknown. To this end, this report provides a comprehensive
  exploration of o1 on different medical scenarios, examining 3 key aspects:
  understanding, reasoning, and multilinguality. Specifically, our evaluation
  encompasses 6 tasks using data from 37 medical datasets, including two newly
  constructed and more challenging question-answering (QA) tasks based on
  professional medical quizzes from the New England Journal of Medicine (NEJM)
  and The Lancet. These datasets offer greater clinical relevance compared to
  standard medical QA benchmarks such as MedQA, translating more effectively into
  real-world clinical utility. Our analysis of o1 suggests that the enhanced
  reasoning ability of LLMs may (significantly) benefit their capability to
  understand various medical instructions and reason through complex clinical
  scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average
  of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios.
  But meanwhile, we identify several weaknesses in both the model capability and
  the existing evaluation protocols, including hallucination, inconsistent
  multilingual ability, and discrepant metrics for evaluation. We release our raw
  data and model outputs at https://ucsc-vlaa.github.io/o1_medicine/ for future
  research.
paper_authors._id: 66f22c153a6d90d35ef1ac71
paper_authors.name: Yunfei Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:44:30.869Z
paper_authors._id: 65f6afa36ea40b9a29dd45ca
paper_authors.avatarUrl: /avatars/a80a25fa70a89a3bf3d50ea2866f0df1.svg
paper_authors.fullname: Yunfei Xie
paper_authors.type: user
paper_authors.user: yunfeixie
paper_authors._id: 66f22c153a6d90d35ef1ac72
paper_authors.name: Juncheng Wu
paper_authors._id: 66f22c153a6d90d35ef1ac73
paper_authors.name: Haoqin Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T08:51:57.148Z
paper_authors._id: 604ae011caabafacfa48e3de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615519738679-noauth.jpeg
paper_authors.fullname: Haoqin Tu
paper_authors.type: user
paper_authors.user: PahaII
paper_authors._id: 66f22c153a6d90d35ef1ac74
paper_authors.name: Siwei Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:17:31.884Z
paper_authors._id: 6433fd1095b8ab049385d2b3
paper_authors.avatarUrl: /avatars/718117cd4631b7c297add7e559032b2f.svg
paper_authors.fullname: Siwei Yang
paper_authors.type: user
paper_authors.user: LAOS-Y
paper_authors._id: 66f22c153a6d90d35ef1ac75
paper_authors.name: Bingchen Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:44.028Z
paper_authors._id: 62dcd71075e9787ec5aa41ba
paper_authors.avatarUrl: /avatars/f37ce036b76180ed0fa004f9c8c09363.svg
paper_authors.fullname: Bingchen Zhao
paper_authors.type: user
paper_authors.user: tennant
paper_authors._id: 66f22c153a6d90d35ef1ac76
paper_authors.name: Yongshuo Zong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-24T07:52:42.234Z
paper_authors._id: 643d5de2c17bfb2256f14b4b
paper_authors.avatarUrl: /avatars/1124128ce69ae6868fd1a2c06420d72f.svg
paper_authors.fullname: Yongshuo Zong
paper_authors.type: user
paper_authors.user: ys-zong
paper_authors._id: 66f22c153a6d90d35ef1ac77
paper_authors.name: Qiao Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-07T16:17:04.847Z
paper_authors._id: 633f51417f69125dcea06aca
paper_authors.avatarUrl: /avatars/2b7fe0d262e336932741d0b498987ae5.svg
paper_authors.fullname: Qiao Jin
paper_authors.type: user
paper_authors.user: qiaojin
paper_authors._id: 66f22c153a6d90d35ef1ac78
paper_authors.name: Cihang Xie
paper_authors._id: 66f22c153a6d90d35ef1ac79
paper_authors.name: Yuyin Zhou
paper_title: A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?
paper_summary: Large language models (LLMs) have exhibited remarkable capabilities across
  various domains and tasks, pushing the boundaries of our knowledge in learning
  and cognition. The latest model, OpenAI's o1, stands out as the first LLM with
  an internalized chain-of-thought technique using reinforcement learning
  strategies. While it has demonstrated surprisingly strong capabilities on
  various general language tasks, its performance in specialized fields such as
  medicine remains unknown. To this end, this report provides a comprehensive
  exploration of o1 on different medical scenarios, examining 3 key aspects:
  understanding, reasoning, and multilinguality. Specifically, our evaluation
  encompasses 6 tasks using data from 37 medical datasets, including two newly
  constructed and more challenging question-answering (QA) tasks based on
  professional medical quizzes from the New England Journal of Medicine (NEJM)
  and The Lancet. These datasets offer greater clinical relevance compared to
  standard medical QA benchmarks such as MedQA, translating more effectively into
  real-world clinical utility. Our analysis of o1 suggests that the enhanced
  reasoning ability of LLMs may (significantly) benefit their capability to
  understand various medical instructions and reason through complex clinical
  scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average
  of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios.
  But meanwhile, we identify several weaknesses in both the model capability and
  the existing evaluation protocols, including hallucination, inconsistent
  multilingual ability, and discrepant metrics for evaluation. We release our raw
  data and model outputs at https://ucsc-vlaa.github.io/o1_medicine/ for future
  research.
paper_ai_summary: OpenAI's o1 demonstrates strong performance in medical tasks but shows weaknesses in reasoning, multilinguality, and evaluation metrics compared to GPT-4.
paper_ai_keywords: chain-of-thought technique reinforcement learning question-answering medical datasets New England Journal of Medicine The Lancet MedQA hallucination inconsistent multilingual ability discrepant metrics