title: S^2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement
    Learning
summary: Recent studies have demonstrated the effectiveness of LLM test-time scaling.
  However, existing approaches to incentivize LLMs' deep thinking abilities
  generally require large-scale data or significant training efforts. Meanwhile,
  it remains unclear how to improve the thinking abilities of less powerful base
  models. In this work, we introduce S^2R, an efficient framework that enhances
  LLM reasoning by teaching models to self-verify and self-correct during
  inference. Specifically, we first initialize LLMs with iterative
  self-verification and self-correction behaviors through supervised fine-tuning
  on carefully curated data. The self-verification and self-correction skills are
  then further strengthened by both outcome-level and process-level reinforcement
  learning, with minimized resource requirements, enabling the model to
  adaptively refine its reasoning process during inference. Our results
  demonstrate that, with only 3.1k self-verifying and self-correcting behavior
  initialization samples, Qwen2.5-math-7B achieves an accuracy improvement from
  51.0\% to 81.6\%, outperforming models trained on an equivalent amount of
  long-CoT distilled data. Extensive experiments and analysis based on three base
  models across both in-domain and out-of-domain benchmarks validate the
  effectiveness of S^2R. Our code and data are available at
  https://github.com/NineAbyss/S2R.
paper_authors._id: 67b69b6717ccb022c6a95b38
paper_authors.name: Ruotian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:58:55.028Z
paper_authors._id: 648294b2eb4befee378951c1
paper_authors.avatarUrl: /avatars/da5d8bf9d8662cc2ffa2c0de49bd66a3.svg
paper_authors.fullname: Ruotian Ma
paper_authors.type: user
paper_authors.user: vvibt
paper_authors._id: 67b69b6717ccb022c6a95b39
paper_authors.name: Peisong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-21T14:59:00.273Z
paper_authors._id: 626f98528a894872cfbf620c
paper_authors.avatarUrl: /avatars/fe31d20313e6ca85e96bc249424c5383.svg
paper_authors.fullname: Peisong Wang
paper_authors.type: user
paper_authors.user: duke1852022
paper_authors._id: 67b69b6717ccb022c6a95b3a
paper_authors.name: Cheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:25:27.094Z
paper_authors._id: 6500234be0c94282ab38cd00
paper_authors.avatarUrl: /avatars/90fc160919cdbb28cfa82becf720b062.svg
paper_authors.fullname: soso
paper_authors.type: user
paper_authors.user: chengliu
paper_authors._id: 67b69b6717ccb022c6a95b3b
paper_authors.name: Xingyan Liu
paper_authors._id: 67b69b6717ccb022c6a95b3c
paper_authors.name: Jiaqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-29T07:59:24.130Z
paper_authors._id: 61b859ddbdf1fac5ed499992
paper_authors.avatarUrl: /avatars/2387fb9b8a46840bfc75248462f0a410.svg
paper_authors.fullname: Jiaqi Chen
paper_authors.type: user
paper_authors.user: judge
paper_authors._id: 67b69b6717ccb022c6a95b3d
paper_authors.name: Bang Zhang
paper_authors._id: 67b69b6717ccb022c6a95b3e
paper_authors.name: Xin Zhou
paper_authors._id: 67b69b6717ccb022c6a95b3f
paper_authors.name: Nan Du
paper_authors._id: 67b69b6717ccb022c6a95b40
paper_authors.name: Jia Li
paper_title: S^2R: Teaching LLMs to Self-verify and Self-correct via Reinforcement
    Learning
paper_summary: Recent studies have demonstrated the effectiveness of LLM test-time scaling.
  However, existing approaches to incentivize LLMs' deep thinking abilities
  generally require large-scale data or significant training efforts. Meanwhile,
  it remains unclear how to improve the thinking abilities of less powerful base
  models. In this work, we introduce S^2R, an efficient framework that enhances
  LLM reasoning by teaching models to self-verify and self-correct during
  inference. Specifically, we first initialize LLMs with iterative
  self-verification and self-correction behaviors through supervised fine-tuning
  on carefully curated data. The self-verification and self-correction skills are
  then further strengthened by both outcome-level and process-level reinforcement
  learning, with minimized resource requirements, enabling the model to
  adaptively refine its reasoning process during inference. Our results
  demonstrate that, with only 3.1k self-verifying and self-correcting behavior
  initialization samples, Qwen2.5-math-7B achieves an accuracy improvement from
  51.0\% to 81.6\%, outperforming models trained on an equivalent amount of
  long-CoT distilled data. Extensive experiments and analysis based on three base
  models across both in-domain and out-of-domain benchmarks validate the
  effectiveness of S^2R. Our code and data are available at
  https://github.com/NineAbyss/S2R.
paper_ai_summary: S$^2$R is a framework that enhances LLM reasoning through iterative self-verification and self-correction using supervised fine-tuning and reinforcement learning with minimal resources.
paper_ai_keywords: LLM self-verification self-correction supervised fine-tuning reinforcement learning Qwen2.5-math-7B long-CoT in-domain out-of-domain benchmarks