Improve Mathematical Reasoning in Language Models by Automated Process
  Supervision
Complex multi-step reasoning tasks, such as solving mathematical problems or
generating code, remain a significant hurdle for even the most advanced large
language models (LLMs). Verifying LLM outputs with an Outcome Reward Model
(ORM) is a standard inference-time technique aimed at enhancing the reasoning
performance of LLMs. However, this still proves insufficient for reasoning
tasks with a lengthy or multi-hop reasoning chain, where the intermediate
outcomes are neither properly rewarded nor penalized. Process supervision
addresses this limitation by assigning intermediate rewards during the
reasoning process. To date, the methods used to collect process supervision
data have relied on either human annotation or per-step Monte Carlo estimation,
both prohibitively expensive to scale, thus hindering the broad application of
this technique. In response to this challenge, we propose a novel
divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named
OmegaPRM for the efficient collection of high-quality process
supervision data. This algorithm swiftly identifies the first error in the
Chain of Thought (CoT) with binary search and balances the positive and
negative examples, thereby ensuring both efficiency and quality. As a result,
we are able to collect over 1.5 million process supervision annotations to
train a Process Reward Model (PRM). Utilizing this fully automated process
supervision alongside the weighted self-consistency algorithm, we have enhanced
the instruction tuned Gemini Pro model's math reasoning performance, achieving
a 69.4\% success rate on the MATH benchmark, a 36\% relative improvement from
the 51\% base model performance. Additionally, the entire process operates
without any human intervention, making our method both financially and
computationally cost-effective compared to existing methods.
6669064673d2031011583c57
Liangchen Luo
admin_assigned
2024-06-12T08:13:24.860Z
645a9ab321ab438e732a7b8d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bNp7XtXqPQoauh7CU5B7b.jpeg
Liangchen Luo
user
luolc
6669064673d2031011583c58
Yinxiao Liu
claimed_verified
2024-06-24T20:18:58.625Z
63449f6acbc4ea2ab7405361
https://cdn-avatars.huggingface.co/v1/production/uploads/1665441596687-noauth.jpeg
Canoee Liu
user
Canoee
6669064673d2031011583c59
Rosanne Liu
6669064673d2031011583c5a
Samrat Phatale
admin_assigned
2024-06-12T08:14:15.496Z
64fb65f348411fc78979a654
/avatars/66b61203f2a6fd07f9026ba97f9e8c5d.svg
Samrat Phatale
user
samratph
6669064673d2031011583c5b
Harsh Lara
admin_assigned
2024-06-12T08:14:21.193Z
64b97f8a008feb6c1ad47fc7
/avatars/f6a0def91074fd079a4a341083c5a0d3.svg
Harsh Lara
user
harrylingua
6669064673d2031011583c5c
Yunxuan Li
admin_assigned
2024-06-12T08:14:32.711Z
63531848fc9b7ad0a53fa564
/avatars/d8df470349aa976741604921cb408440.svg
Yunxuan Li
user
eric008
6669064673d2031011583c5d
Lei Shu
6669064673d2031011583c5e
Yun Zhu
6669064673d2031011583c5f
Lei Meng
6669064673d2031011583c60
Jiao Sun
admin_assigned
2024-06-12T08:15:08.371Z
642cd2f7ad221e8f41d2f05e
/avatars/64ad33b67c10e3cb7b9b07744d108d8c.svg
Jiao Sun
user
sunjiao123sun
6669064673d2031011583c61
Abhinav Rastogi
admin_assigned
2024-06-12T08:15:35.463Z
653966932a101a151da1155c
/avatars/10baa6536793b397c05028011d4ffcdc.svg
Abhinav Rastogi
user
abhirast
Improve Mathematical Reasoning in Language Models by Automated Process
  Supervision
Complex multi-step reasoning tasks, such as solving mathematical problems or
generating code, remain a significant hurdle for even the most advanced large
language models (LLMs). Verifying LLM outputs with an Outcome Reward Model
(ORM) is a standard inference-time technique aimed at enhancing the reasoning
performance of LLMs. However, this still proves insufficient for reasoning
tasks with a lengthy or multi-hop reasoning chain, where the intermediate
outcomes are neither properly rewarded nor penalized. Process supervision
addresses this limitation by assigning intermediate rewards during the
reasoning process. To date, the methods used to collect process supervision
data have relied on either human annotation or per-step Monte Carlo estimation,
both prohibitively expensive to scale, thus hindering the broad application of
this technique. In response to this challenge, we propose a novel
divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named
OmegaPRM for the efficient collection of high-quality process
supervision data. This algorithm swiftly identifies the first error in the
Chain of Thought (CoT) with binary search and balances the positive and
negative examples, thereby ensuring both efficiency and quality. As a result,
we are able to collect over 1.5 million process supervision annotations to
train a Process Reward Model (PRM). Utilizing this fully automated process
supervision alongside the weighted self-consistency algorithm, we have enhanced
the instruction tuned Gemini Pro model's math reasoning performance, achieving
a 69.4\% success rate on the MATH benchmark, a 36\% relative improvement from
the 51\% base model performance. Additionally, the entire process operates
without any human intervention, making our method both financially and
computationally cost-effective compared to existing methods.
A novel divide-and-conquer Monte Carlo Tree Search algorithm called OmegaPRM enables efficient collection of process supervision data to enhance the reasoning performance of LLMs without human intervention.
Outcome Reward Model Process supervision Monte Carlo Tree Search OmegaPRM Chain of Thought Process Reward Model weighted self-consistency