title: Self-rewarding correction for mathematical reasoning
summary: We study self-rewarding reasoning large language models (LLMs), which can
  simultaneously generate step-by-step reasoning and evaluate the correctness of
  their outputs during the inference time-without external feedback. This
  integrated approach allows a single model to independently guide its reasoning
  process, offering computational advantages for model deployment. We
  particularly focus on the representative task of self-correction, where models
  autonomously detect errors in their responses, revise outputs, and decide when
  to terminate iterative refinement loops. To enable this, we propose a
  two-staged algorithmic framework for constructing self-rewarding reasoning
  models using only self-generated data. In the first stage, we employ sequential
  rejection sampling to synthesize long chain-of-thought trajectories that
  incorporate both self-rewarding and self-correction mechanisms. Fine-tuning
  models on these curated data allows them to learn the patterns of
  self-rewarding and self-correction. In the second stage, we further enhance the
  models' ability to assess response accuracy and refine outputs through
  reinforcement learning with rule-based signals. Experiments with Llama-3 and
  Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction
  capabilities and achieves performance comparable to systems that rely on
  external reward models.
paper_authors._id: 67c12987505a88e4a185e0d7
paper_authors.name: Wei Xiong
paper_authors._id: 67c12987505a88e4a185e0d8
paper_authors.name: Hanning Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:22:33.128Z
paper_authors._id: 6470e0f1cfd57849519033a5
paper_authors.avatarUrl: /avatars/7ffefee3e36a4e37b9f4510bc6b689d1.svg
paper_authors.fullname: Hanning Zhang
paper_authors.type: user
paper_authors.user: HanningZhang
paper_authors._id: 67c12987505a88e4a185e0d9
paper_authors.name: Chenlu Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:22:38.981Z
paper_authors._id: 65eec5c1d7d63c2ed0615421
paper_authors.avatarUrl: /avatars/8c32f5e7d4b1940088bdec73c0b86fab.svg
paper_authors.fullname: Chenlu Ye
paper_authors.type: user
paper_authors.user: Chenlu123
paper_authors._id: 67c12987505a88e4a185e0da
paper_authors.name: Lichang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:14:29.479Z
paper_authors._id: 62323bb408bcea92917e42ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
paper_authors.fullname: Lichang Chen
paper_authors.type: user
paper_authors.user: Lichang-Chen
paper_authors._id: 67c12987505a88e4a185e0db
paper_authors.name: Nan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:23:02.992Z
paper_authors._id: 64b8922ca1827cc8d04ae919
paper_authors.avatarUrl: /avatars/0aaa83e3d09a82434e1d6af724aaa485.svg
paper_authors.fullname: Nan Jiang
paper_authors.type: user
paper_authors.user: nanjiang
paper_authors._id: 67c12987505a88e4a185e0dc
paper_authors.name: Tong Zhang
paper_title: Self-rewarding correction for mathematical reasoning
paper_summary: We study self-rewarding reasoning large language models (LLMs), which can
  simultaneously generate step-by-step reasoning and evaluate the correctness of
  their outputs during the inference time-without external feedback. This
  integrated approach allows a single model to independently guide its reasoning
  process, offering computational advantages for model deployment. We
  particularly focus on the representative task of self-correction, where models
  autonomously detect errors in their responses, revise outputs, and decide when
  to terminate iterative refinement loops. To enable this, we propose a
  two-staged algorithmic framework for constructing self-rewarding reasoning
  models using only self-generated data. In the first stage, we employ sequential
  rejection sampling to synthesize long chain-of-thought trajectories that
  incorporate both self-rewarding and self-correction mechanisms. Fine-tuning
  models on these curated data allows them to learn the patterns of
  self-rewarding and self-correction. In the second stage, we further enhance the
  models' ability to assess response accuracy and refine outputs through
  reinforcement learning with rule-based signals. Experiments with Llama-3 and
  Qwen-2.5 demonstrate that our approach surpasses intrinsic self-correction
  capabilities and achieves performance comparable to systems that rely on
  external reward models.
paper_ai_keywords: self-rewarding reasoning large language models (LLMs) step-by-step reasoning inference time self-correction reasoning process two-staged algorithmic framework sequential rejection sampling chain-of-thought trajectories self-rewarding self-correction mechanisms fine-tuning reinforcement learning rule-based signals intrinsic self-correction external reward models