title: Improve Mathematical Reasoning in Language Models by Automated Process
    Supervision
summary: Complex multi-step reasoning tasks, such as solving mathematical problems or
  generating code, remain a significant hurdle for even the most advanced large
  language models (LLMs). Verifying LLM outputs with an Outcome Reward Model
  (ORM) is a standard inference-time technique aimed at enhancing the reasoning
  performance of LLMs. However, this still proves insufficient for reasoning
  tasks with a lengthy or multi-hop reasoning chain, where the intermediate
  outcomes are neither properly rewarded nor penalized. Process supervision
  addresses this limitation by assigning intermediate rewards during the
  reasoning process. To date, the methods used to collect process supervision
  data have relied on either human annotation or per-step Monte Carlo estimation,
  both prohibitively expensive to scale, thus hindering the broad application of
  this technique. In response to this challenge, we propose a novel
  divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named
  OmegaPRM for the efficient collection of high-quality process
  supervision data. This algorithm swiftly identifies the first error in the
  Chain of Thought (CoT) with binary search and balances the positive and
  negative examples, thereby ensuring both efficiency and quality. As a result,
  we are able to collect over 1.5 million process supervision annotations to
  train a Process Reward Model (PRM). Utilizing this fully automated process
  supervision alongside the weighted self-consistency algorithm, we have enhanced
  the instruction tuned Gemini Pro model's math reasoning performance, achieving
  a 69.4\% success rate on the MATH benchmark, a 36\% relative improvement from
  the 51\% base model performance. Additionally, the entire process operates
  without any human intervention, making our method both financially and
  computationally cost-effective compared to existing methods.
paper_authors._id: 6669064673d2031011583c57
paper_authors.name: Liangchen Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:13:24.860Z
paper_authors._id: 645a9ab321ab438e732a7b8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/bNp7XtXqPQoauh7CU5B7b.jpeg
paper_authors.fullname: Liangchen Luo
paper_authors.type: user
paper_authors.user: luolc
paper_authors._id: 6669064673d2031011583c58
paper_authors.name: Yinxiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T20:18:58.625Z
paper_authors._id: 63449f6acbc4ea2ab7405361
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665441596687-noauth.jpeg
paper_authors.fullname: Canoee Liu
paper_authors.type: user
paper_authors.user: Canoee
paper_authors._id: 6669064673d2031011583c59
paper_authors.name: Rosanne Liu
paper_authors._id: 6669064673d2031011583c5a
paper_authors.name: Samrat Phatale
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:14:15.496Z
paper_authors._id: 64fb65f348411fc78979a654
paper_authors.avatarUrl: /avatars/66b61203f2a6fd07f9026ba97f9e8c5d.svg
paper_authors.fullname: Samrat Phatale
paper_authors.type: user
paper_authors.user: samratph
paper_authors._id: 6669064673d2031011583c5b
paper_authors.name: Harsh Lara
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:14:21.193Z
paper_authors._id: 64b97f8a008feb6c1ad47fc7
paper_authors.avatarUrl: /avatars/f6a0def91074fd079a4a341083c5a0d3.svg
paper_authors.fullname: Harsh Lara
paper_authors.type: user
paper_authors.user: harrylingua
paper_authors._id: 6669064673d2031011583c5c
paper_authors.name: Yunxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:14:32.711Z
paper_authors._id: 63531848fc9b7ad0a53fa564
paper_authors.avatarUrl: /avatars/d8df470349aa976741604921cb408440.svg
paper_authors.fullname: Yunxuan Li
paper_authors.type: user
paper_authors.user: eric008
paper_authors._id: 6669064673d2031011583c5d
paper_authors.name: Lei Shu
paper_authors._id: 6669064673d2031011583c5e
paper_authors.name: Yun Zhu
paper_authors._id: 6669064673d2031011583c5f
paper_authors.name: Lei Meng
paper_authors._id: 6669064673d2031011583c60
paper_authors.name: Jiao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:15:08.371Z
paper_authors._id: 642cd2f7ad221e8f41d2f05e
paper_authors.avatarUrl: /avatars/64ad33b67c10e3cb7b9b07744d108d8c.svg
paper_authors.fullname: Jiao Sun
paper_authors.type: user
paper_authors.user: sunjiao123sun
paper_authors._id: 6669064673d2031011583c61
paper_authors.name: Abhinav Rastogi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T08:15:35.463Z
paper_authors._id: 653966932a101a151da1155c
paper_authors.avatarUrl: /avatars/10baa6536793b397c05028011d4ffcdc.svg
paper_authors.fullname: Abhinav Rastogi
paper_authors.type: user
paper_authors.user: abhirast
paper_title: Improve Mathematical Reasoning in Language Models by Automated Process
    Supervision
paper_summary: Complex multi-step reasoning tasks, such as solving mathematical problems or
  generating code, remain a significant hurdle for even the most advanced large
  language models (LLMs). Verifying LLM outputs with an Outcome Reward Model
  (ORM) is a standard inference-time technique aimed at enhancing the reasoning
  performance of LLMs. However, this still proves insufficient for reasoning
  tasks with a lengthy or multi-hop reasoning chain, where the intermediate
  outcomes are neither properly rewarded nor penalized. Process supervision
  addresses this limitation by assigning intermediate rewards during the
  reasoning process. To date, the methods used to collect process supervision
  data have relied on either human annotation or per-step Monte Carlo estimation,
  both prohibitively expensive to scale, thus hindering the broad application of
  this technique. In response to this challenge, we propose a novel
  divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named
  OmegaPRM for the efficient collection of high-quality process
  supervision data. This algorithm swiftly identifies the first error in the
  Chain of Thought (CoT) with binary search and balances the positive and
  negative examples, thereby ensuring both efficiency and quality. As a result,
  we are able to collect over 1.5 million process supervision annotations to
  train a Process Reward Model (PRM). Utilizing this fully automated process
  supervision alongside the weighted self-consistency algorithm, we have enhanced
  the instruction tuned Gemini Pro model's math reasoning performance, achieving
  a 69.4\% success rate on the MATH benchmark, a 36\% relative improvement from
  the 51\% base model performance. Additionally, the entire process operates
  without any human intervention, making our method both financially and
  computationally cost-effective compared to existing methods.
paper_ai_keywords: Outcome Reward Model Process supervision Monte Carlo Tree Search (MCTS) OmegaPRM Chain of Thought (CoT) Process Reward Model (PRM) weighted self-consistency algorithm MATH benchmark