title: From Medprompt to o1: Exploration of Run-Time Strategies for Medical
    Challenge Problems and Beyond
summary: Run-time steering strategies like Medprompt are valuable for guiding large
  language models (LLMs) to top performance on challenging tasks. Medprompt
  demonstrates that a general LLM can be focused to deliver state-of-the-art
  performance on specialized domains like medicine by using a prompt to elicit a
  run-time strategy involving chain of thought reasoning and ensembling. OpenAI's
  o1-preview model represents a new paradigm, where a model is designed to do
  run-time reasoning before generating final responses. We seek to understand the
  behavior of o1-preview on a diverse set of medical challenge problem
  benchmarks. Following on the Medprompt study with GPT-4, we systematically
  evaluate the o1-preview model across various medical benchmarks. Notably, even
  without prompting techniques, o1-preview largely outperforms the GPT-4 series
  with Medprompt. We further systematically study the efficacy of classic prompt
  engineering strategies, as represented by Medprompt, within the new paradigm of
  reasoning models. We found that few-shot prompting hinders o1's performance,
  suggesting that in-context learning may no longer be an effective steering
  approach for reasoning-native models. While ensembling remains viable, it is
  resource-intensive and requires careful cost-performance optimization. Our cost
  and accuracy analysis across run-time strategies reveals a Pareto frontier,
  with GPT-4o representing a more affordable option and o1-preview achieving
  state-of-the-art performance at higher cost. Although o1-preview offers top
  performance, GPT-4o with steering strategies like Medprompt retains value in
  specific contexts. Moreover, we note that the o1-preview model has reached
  near-saturation on many existing medical benchmarks, underscoring the need for
  new, challenging benchmarks. We close with reflections on general directions
  for inference-time computation with LLMs.
paper_authors._id: 672cedbe75014b062734d2ff
paper_authors.name: Harsha Nori
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-11-07T16:41:35.821Z
paper_authors._id: 63336c425a032dcd095ba3fe
paper_authors.avatarUrl: /avatars/72ce38aa86b4ada769539f0d91ce6937.svg
paper_authors.fullname: Harsha Nori
paper_authors.type: user
paper_authors.user: hanori
paper_authors._id: 672cedbe75014b062734d300
paper_authors.name: Naoto Usuyama
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:32:49.980Z
paper_authors._id: 5f243561e923d665e616263f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1596208460315-noauth.jpeg
paper_authors.fullname: Naoto Usuyama
paper_authors.type: user
paper_authors.user: naotous
paper_authors._id: 672cedbe75014b062734d301
paper_authors.name: Nicholas King
paper_authors._id: 672cedbe75014b062734d302
paper_authors.name: Scott Mayer McKinney
paper_authors._id: 672cedbe75014b062734d303
paper_authors.name: Xavier Fernandes
paper_authors._id: 672cedbe75014b062734d304
paper_authors.name: Sheng Zhang
paper_authors._id: 672cedbe75014b062734d305
paper_authors.name: Eric Horvitz
paper_title: From Medprompt to o1: Exploration of Run-Time Strategies for Medical
    Challenge Problems and Beyond
paper_summary: Run-time steering strategies like Medprompt are valuable for guiding large
  language models (LLMs) to top performance on challenging tasks. Medprompt
  demonstrates that a general LLM can be focused to deliver state-of-the-art
  performance on specialized domains like medicine by using a prompt to elicit a
  run-time strategy involving chain of thought reasoning and ensembling. OpenAI's
  o1-preview model represents a new paradigm, where a model is designed to do
  run-time reasoning before generating final responses. We seek to understand the
  behavior of o1-preview on a diverse set of medical challenge problem
  benchmarks. Following on the Medprompt study with GPT-4, we systematically
  evaluate the o1-preview model across various medical benchmarks. Notably, even
  without prompting techniques, o1-preview largely outperforms the GPT-4 series
  with Medprompt. We further systematically study the efficacy of classic prompt
  engineering strategies, as represented by Medprompt, within the new paradigm of
  reasoning models. We found that few-shot prompting hinders o1's performance,
  suggesting that in-context learning may no longer be an effective steering
  approach for reasoning-native models. While ensembling remains viable, it is
  resource-intensive and requires careful cost-performance optimization. Our cost
  and accuracy analysis across run-time strategies reveals a Pareto frontier,
  with GPT-4o representing a more affordable option and o1-preview achieving
  state-of-the-art performance at higher cost. Although o1-preview offers top
  performance, GPT-4o with steering strategies like Medprompt retains value in
  specific contexts. Moreover, we note that the o1-preview model has reached
  near-saturation on many existing medical benchmarks, underscoring the need for
  new, challenging benchmarks. We close with reflections on general directions
  for inference-time computation with LLMs.