title: GLoRe: When, Where, and How to Improve LLM Reasoning via Global and
    Local Refinements
summary: State-of-the-art language models can exhibit impressive reasoning refinement
  capabilities on math, science or coding tasks. However, recent work
  demonstrates that even the best models struggle to identify when and
  where to refine without access to external feedback. Outcome-based Reward
  Models (ORMs), trained to predict correctness of the final answer
  indicating when to refine, offer one convenient solution for deciding when to
  refine. Process Based Reward Models (PRMs), trained to predict
  correctness of intermediate steps, can then be used to indicate where to
  refine. But they are expensive to train, requiring extensive human annotations.
  In this paper, we propose Stepwise ORMs (SORMs) which are trained,
  only on synthetic data, to approximate the expected future reward of the
  optimal policy or V^{star}. More specifically, SORMs are trained to predict
  the correctness of the final answer when sampling the current policy many times
  (rather than only once as in the case of ORMs). Our experiments show that SORMs
  can more accurately detect incorrect reasoning steps compared to ORMs, thus
  improving downstream accuracy when doing refinements. We then train
  global refinement models, which take only the question and a draft
  solution as input and predict a corrected solution, and local
  refinement models which also take as input a critique indicating the location
  of the first reasoning error. We generate training data for both models
  synthetically by reusing data used to train the SORM. We find combining global
  and local refinements, using the ORM as a reranker, significantly outperforms
  either one individually, as well as a best of three sample baseline. With this
  strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned
  with RL) on GSM8K from 53\% to 65\% when greedily sampled.
paper_authors._id: 65d430f7bd2ebbf302102aab
paper_authors.name: Alex Havrilla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:47:33.707Z
paper_authors._id: 6144d541fa30a4ba3ef7ef15
paper_authors.avatarUrl: /avatars/eeb849eb29e249fd1b40e5c10d9bf9e5.svg
paper_authors.fullname: Alex Havrilla
paper_authors.type: user
paper_authors.user: Dahoas
paper_authors._id: 65d430f7bd2ebbf302102aac
paper_authors.name: Sharath Raparthy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:47:41.618Z
paper_authors._id: 65007a96ea5699b59a9bb432
paper_authors.avatarUrl: /avatars/4549f5964b007de189c45a77daf10075.svg
paper_authors.fullname: Sharath Raparthy
paper_authors.type: user
paper_authors.user: sharathraparthy
paper_authors._id: 65d430f7bd2ebbf302102aad
paper_authors.name: Christoforus Nalmpantis
paper_authors._id: 65d430f7bd2ebbf302102aae
paper_authors.name: Jane Dwivedi-Yu
paper_authors._id: 65d430f7bd2ebbf302102aaf
paper_authors.name: Maksym Zhuravinskyi
paper_authors._id: 65d430f7bd2ebbf302102ab0
paper_authors.name: Eric Hambro
paper_authors._id: 65d430f7bd2ebbf302102ab1
paper_authors.name: Roberta Railneau
paper_title: GLoRe: When, Where, and How to Improve LLM Reasoning via Global and
    Local Refinements
paper_summary: State-of-the-art language models can exhibit impressive reasoning refinement
  capabilities on math, science or coding tasks. However, recent work
  demonstrates that even the best models struggle to identify when and
  where to refine without access to external feedback. Outcome-based Reward
  Models (ORMs), trained to predict correctness of the final answer
  indicating when to refine, offer one convenient solution for deciding when to
  refine. Process Based Reward Models (PRMs), trained to predict
  correctness of intermediate steps, can then be used to indicate where to
  refine. But they are expensive to train, requiring extensive human annotations.
  In this paper, we propose Stepwise ORMs (SORMs) which are trained,
  only on synthetic data, to approximate the expected future reward of the
  optimal policy or V^{star}. More specifically, SORMs are trained to predict
  the correctness of the final answer when sampling the current policy many times
  (rather than only once as in the case of ORMs). Our experiments show that SORMs
  can more accurately detect incorrect reasoning steps compared to ORMs, thus
  improving downstream accuracy when doing refinements. We then train
  global refinement models, which take only the question and a draft
  solution as input and predict a corrected solution, and local
  refinement models which also take as input a critique indicating the location
  of the first reasoning error. We generate training data for both models
  synthetically by reusing data used to train the SORM. We find combining global
  and local refinements, using the ORM as a reranker, significantly outperforms
  either one individually, as well as a best of three sample baseline. With this
  strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned
  with RL) on GSM8K from 53\% to 65\% when greedily sampled.
paper_ai_summary: Stepwise ORMs and combinations of global and local refinement models improve reasoning accuracy in state-of-the-art language models by leveraging synthetic data and ORMs for better error detection and correction.
paper_ai_keywords: language models reasoning refinement math science coding tasks Outcome-based Reward Models (ORMs) Process Based Reward Models (PRMs) Stepwise ORMs (SORMs) synthetic data expected future reward optimal policy global refinement models local refinement models GSM8K LLaMA-2 13B