title: The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in
    Learning to Reason
summary: Recent studies on post-training large language models (LLMs) for reasoning
  through reinforcement learning (RL) typically focus on tasks that can be
  accurately verified and rewarded, such as solving math problems. In contrast,
  our research investigates the impact of reward noise, a more practical
  consideration for real-world scenarios involving the post-training of LLMs
  using reward models. We found that LLMs demonstrate strong robustness to
  substantial reward noise. For example, manually flipping 40% of the reward
  function's outputs in math tasks still allows a Qwen-2.5-7B model to achieve
  rapid convergence, improving its performance on math tasks from 5% to 72%,
  compared to the 75% accuracy achieved by a model trained with noiseless
  rewards. Surprisingly, by only rewarding the appearance of key reasoning
  phrases (namely reasoning pattern reward, RPR), such as ``first, I need
  to''-without verifying the correctness of answers, the model achieved peak
  downstream performance (over 70% accuracy for Qwen-2.5-7B) comparable to models
  trained with strict correctness verification and accurate rewards. Recognizing
  the importance of the reasoning process over the final results, we combined RPR
  with noisy reward models. RPR helped calibrate the noisy reward models,
  mitigating potential false negatives and enhancing the LLM's performance on
  open-ended tasks. These findings suggest the importance of improving models'
  foundational abilities during the pre-training phase while providing insights
  for advancing post-training techniques. Our code and scripts are available at
  https://github.com/trestad/Noisy-Rewards-in-Learning-to-Reason.
paper_authors._id: 6838bb282b382ba50bdcddc4
paper_authors.name: Ang Lv
paper_authors._id: 6838bb282b382ba50bdcddc5
paper_authors.name: Ruobing Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:45.319Z
paper_authors._id: 6622443b9b0614a760dd8123
paper_authors.avatarUrl: /avatars/acb6c1c9c429af1112530dcf76a8e420.svg
paper_authors.fullname: Ruobing Xie
paper_authors.type: user
paper_authors.user: Ruobing-Xie
paper_authors._id: 6838bb282b382ba50bdcddc6
paper_authors.name: Xingwu Sun
paper_authors._id: 6838bb282b382ba50bdcddc7
paper_authors.name: Zhanhui Kang
paper_authors._id: 6838bb282b382ba50bdcddc8
paper_authors.name: Rui Yan
paper_title: The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in
    Learning to Reason
paper_summary: Recent studies on post-training large language models (LLMs) for reasoning
  through reinforcement learning (RL) typically focus on tasks that can be
  accurately verified and rewarded, such as solving math problems. In contrast,
  our research investigates the impact of reward noise, a more practical
  consideration for real-world scenarios involving the post-training of LLMs
  using reward models. We found that LLMs demonstrate strong robustness to
  substantial reward noise. For example, manually flipping 40% of the reward
  function's outputs in math tasks still allows a Qwen-2.5-7B model to achieve
  rapid convergence, improving its performance on math tasks from 5% to 72%,
  compared to the 75% accuracy achieved by a model trained with noiseless
  rewards. Surprisingly, by only rewarding the appearance of key reasoning
  phrases (namely reasoning pattern reward, RPR), such as ``first, I need
  to''-without verifying the correctness of answers, the model achieved peak
  downstream performance (over 70% accuracy for Qwen-2.5-7B) comparable to models
  trained with strict correctness verification and accurate rewards. Recognizing
  the importance of the reasoning process over the final results, we combined RPR
  with noisy reward models. RPR helped calibrate the noisy reward models,
  mitigating potential false negatives and enhancing the LLM's performance on
  open-ended tasks. These findings suggest the importance of improving models'
  foundational abilities during the pre-training phase while providing insights
  for advancing post-training techniques. Our code and scripts are available at
  https://github.com/trestad/Noisy-Rewards-in-Learning-to-Reason.
paper_ai_summary: LLMs exhibit robustness to reward noise during post-training and achieve high performance using reasoning pattern rewards (RPR) in conjunction with noisy reward models.
paper_ai_keywords: large language models (LLMs) post-training reinforcement learning (RL) reward noise reward models rapid convergence reasoning pattern reward (RPR) false negatives open-ended tasks