GLoRe: When, Where, and How to Improve LLM Reasoning via Global and
  Local Refinements
State-of-the-art language models can exhibit impressive reasoning refinement
capabilities on math, science or coding tasks. However, recent work
demonstrates that even the best models struggle to identify when and
where to refine without access to external feedback. Outcome-based Reward
Models (ORMs), trained to predict correctness of the final answer
indicating when to refine, offer one convenient solution for deciding when to
refine. Process Based Reward Models (PRMs), trained to predict
correctness of intermediate steps, can then be used to indicate where to
refine. But they are expensive to train, requiring extensive human annotations.
In this paper, we propose Stepwise ORMs (SORMs) which are trained,
only on synthetic data, to approximate the expected future reward of the
optimal policy or V^{star}. More specifically, SORMs are trained to predict
the correctness of the final answer when sampling the current policy many times
(rather than only once as in the case of ORMs). Our experiments show that SORMs
can more accurately detect incorrect reasoning steps compared to ORMs, thus
improving downstream accuracy when doing refinements. We then train
global refinement models, which take only the question and a draft
solution as input and predict a corrected solution, and local
refinement models which also take as input a critique indicating the location
of the first reasoning error. We generate training data for both models
synthetically by reusing data used to train the SORM. We find combining global
and local refinements, using the ORM as a reranker, significantly outperforms
either one individually, as well as a best of three sample baseline. With this
strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned
with RL) on GSM8K from 53\% to 65\% when greedily sampled.
65d430f7bd2ebbf302102aab
Alex Havrilla
admin_assigned
2024-02-20T09:47:33.707Z
6144d541fa30a4ba3ef7ef15
/avatars/eeb849eb29e249fd1b40e5c10d9bf9e5.svg
Alex Havrilla
user
Dahoas
65d430f7bd2ebbf302102aac
Sharath Raparthy
admin_assigned
2024-02-20T09:47:41.618Z
65007a96ea5699b59a9bb432
/avatars/4549f5964b007de189c45a77daf10075.svg
Sharath Raparthy
user
sharathraparthy
65d430f7bd2ebbf302102aad
Christoforus Nalmpantis
65d430f7bd2ebbf302102aae
Jane Dwivedi-Yu
65d430f7bd2ebbf302102aaf
Maksym Zhuravinskyi
65d430f7bd2ebbf302102ab0
Eric Hambro
65d430f7bd2ebbf302102ab1
Roberta Railneau
GLoRe: When, Where, and How to Improve LLM Reasoning via Global and
  Local Refinements
State-of-the-art language models can exhibit impressive reasoning refinement
capabilities on math, science or coding tasks. However, recent work
demonstrates that even the best models struggle to identify when and
where to refine without access to external feedback. Outcome-based Reward
Models (ORMs), trained to predict correctness of the final answer
indicating when to refine, offer one convenient solution for deciding when to
refine. Process Based Reward Models (PRMs), trained to predict
correctness of intermediate steps, can then be used to indicate where to
refine. But they are expensive to train, requiring extensive human annotations.
In this paper, we propose Stepwise ORMs (SORMs) which are trained,
only on synthetic data, to approximate the expected future reward of the
optimal policy or V^{star}. More specifically, SORMs are trained to predict
the correctness of the final answer when sampling the current policy many times
(rather than only once as in the case of ORMs). Our experiments show that SORMs
can more accurately detect incorrect reasoning steps compared to ORMs, thus
improving downstream accuracy when doing refinements. We then train
global refinement models, which take only the question and a draft
solution as input and predict a corrected solution, and local
refinement models which also take as input a critique indicating the location
of the first reasoning error. We generate training data for both models
synthetically by reusing data used to train the SORM. We find combining global
and local refinements, using the ORM as a reranker, significantly outperforms
either one individually, as well as a best of three sample baseline. With this
strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned
with RL) on GSM8K from 53\% to 65\% when greedily sampled.
Stepwise ORMs and combinations of global and local refinement models improve reasoning accuracy in state-of-the-art language models by leveraging synthetic data and ORMs for better error detection and correction.
language models reasoning refinement math science coding tasks Outcome-based Reward Models (ORMs) Process Based Reward Models (PRMs) Stepwise ORMs (SORMs) synthetic data expected future reward optimal policy global refinement models local refinement models GSM8K LLaMA-2 13B