title: Exploring the Limit of Outcome Reward for Learning Mathematical
    Reasoning
summary: Reasoning abilities, especially those for solving complex math problems, are
  crucial components of general intelligence. Recent advances by proprietary
  companies, such as o-series models of OpenAI, have made remarkable progress on
  reasoning tasks. However, the complete technical details remain unrevealed, and
  the techniques that are believed certainly to be adopted are only reinforcement
  learning (RL) and the long chain of thoughts. This paper proposes a new RL
  framework, termed OREAL, to pursue the performance limit that can be achieved
  through Outcome REwArd-based reinforcement
  Learning for mathematical reasoning tasks, where only binary outcome
  rewards are easily accessible. We theoretically prove that behavior cloning on
  positive trajectories from best-of-N (BoN) sampling is sufficient to learn the
  KL-regularized optimal policy in binary feedback environments. This formulation
  further implies that the rewards of negative samples should be reshaped to
  ensure the gradient consistency between positive and negative samples. To
  alleviate the long-existing difficulties brought by sparse rewards in RL, which
  are even exacerbated by the partial correctness of the long chain of thought
  for reasoning tasks, we further apply a token-level reward model to sample
  important tokens in reasoning trajectories for learning. With OREAL, for the
  first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL,
  being on par with 32B models. OREAL-32B also surpasses previous 32B models
  trained by distillation with 95.0 pass@1 accuracy on MATH-500. Our
  investigation also indicates the importance of initial policy models and
  training queries for RL. Code, models, and data will be released to benefit
  future researchhttps://github.com/InternLM/OREAL.
paper_authors._id: 67aacd7e078cdf445284f9f6
paper_authors.name: Chengqi Lyu
paper_authors._id: 67aacd7e078cdf445284f9f7
paper_authors.name: Songyang Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-11T15:56:04.271Z
paper_authors._id: 650ab54e23196fb2d86b486b
paper_authors.avatarUrl: /avatars/e0506393589695b553ec9ee3fe99b93a.svg
paper_authors.fullname: SongYang Gao
paper_authors.type: user
paper_authors.user: Wizardcoast
paper_authors._id: 67aacd7e078cdf445284f9f8
paper_authors.name: Yuzhe Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T16:40:46.962Z
paper_authors._id: 6601196cc91ba4c08ad6e270
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6601196cc91ba4c08ad6e270/X2YPNzUOQXBz5Gv-xR9LW.jpeg
paper_authors.fullname: yuzhe gu
paper_authors.type: user
paper_authors.user: vanilla1116
paper_authors._id: 67aacd7e078cdf445284f9f9
paper_authors.name: Wenwei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:54:40.279Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 67aacd7e078cdf445284f9fa
paper_authors.name: Jianfei Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-11T15:56:31.173Z
paper_authors._id: 64070c5c4dc5f2846c925e93
paper_authors.avatarUrl: /avatars/ac2d7c1cd4ecccd6a88b85767c963ec7.svg
paper_authors.fullname: Gao Jianfei
paper_authors.type: user
paper_authors.user: pppppM
paper_authors._id: 67aacd7e078cdf445284f9fb
paper_authors.name: Kuikun Liu
paper_authors._id: 67aacd7e078cdf445284f9fc
paper_authors.name: Ziyi Wang
paper_authors._id: 67aacd7e078cdf445284f9fd
paper_authors.name: Shuaibin Li
paper_authors._id: 67aacd7e078cdf445284f9fe
paper_authors.name: Qian Zhao
paper_authors._id: 67aacd7e078cdf445284f9ff
paper_authors.name: Haian Huang
paper_authors._id: 67aacd7e078cdf445284fa00
paper_authors.name: Weihan Cao
paper_authors._id: 67aacd7e078cdf445284fa01
paper_authors.name: Jiangning Liu
paper_authors._id: 67aacd7e078cdf445284fa02
paper_authors.name: Hongwei Liu
paper_authors._id: 67aacd7e078cdf445284fa03
paper_authors.name: Junnan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:56:53.050Z
paper_authors._id: 643d26979347842571bc9613
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3heFf7h3jbhhJWJ4JfGfh.jpeg
paper_authors.fullname: Junnan Liu
paper_authors.type: user
paper_authors.user: jnanliu
paper_authors._id: 67aacd7e078cdf445284fa04
paper_authors.name: Songyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:54:37.733Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 67aacd7e078cdf445284fa05
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T16:16:07.907Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 67aacd7e078cdf445284fa06
paper_authors.name: Kai Chen
paper_title: Exploring the Limit of Outcome Reward for Learning Mathematical
    Reasoning
paper_summary: Reasoning abilities, especially those for solving complex math problems, are
  crucial components of general intelligence. Recent advances by proprietary
  companies, such as o-series models of OpenAI, have made remarkable progress on
  reasoning tasks. However, the complete technical details remain unrevealed, and
  the techniques that are believed certainly to be adopted are only reinforcement
  learning (RL) and the long chain of thoughts. This paper proposes a new RL
  framework, termed OREAL, to pursue the performance limit that can be achieved
  through Outcome REwArd-based reinforcement
  Learning for mathematical reasoning tasks, where only binary outcome
  rewards are easily accessible. We theoretically prove that behavior cloning on
  positive trajectories from best-of-N (BoN) sampling is sufficient to learn the
  KL-regularized optimal policy in binary feedback environments. This formulation
  further implies that the rewards of negative samples should be reshaped to
  ensure the gradient consistency between positive and negative samples. To
  alleviate the long-existing difficulties brought by sparse rewards in RL, which
  are even exacerbated by the partial correctness of the long chain of thought
  for reasoning tasks, we further apply a token-level reward model to sample
  important tokens in reasoning trajectories for learning. With OREAL, for the
  first time, a 7B model can obtain 94.0 pass@1 accuracy on MATH-500 through RL,
  being on par with 32B models. OREAL-32B also surpasses previous 32B models
  trained by distillation with 95.0 pass@1 accuracy on MATH-500. Our
  investigation also indicates the importance of initial policy models and
  training queries for RL. Code, models, and data will be released to benefit
  future researchhttps://github.com/InternLM/OREAL.
paper_ai_summary: A novel reinforcement learning framework, OREAL, using outcome rewards achieves high accuracy in mathematical reasoning tasks with smaller models compared to larger models trained using distillation.
paper_ai_keywords: reinforcement learning OREAL Outcome REwArd-based reinforcement Learning binary outcome rewards behavior cloning best-of-N KL-regularized optimal policy token-level reward model MATH-500 pass@1 accuracy