title: Training Language Models to Self-Correct via Reinforcement Learning
summary: Self-correction is a highly desirable capability of large language models
  (LLMs), yet it has consistently been found to be largely ineffective in modern
  LLMs. Existing approaches for training self-correction either require multiple
  models or rely on a more capable model or other forms of supervision. To this
  end, we develop a multi-turn online reinforcement learning (RL) approach,
  SCoRe, that significantly improves an LLM's self-correction ability using
  entirely self-generated data. To build SCoRe, we first show that variants of
  supervised fine-tuning (SFT) on offline model-generated correction traces are
  insufficient for instilling self-correction behavior. In particular, we observe
  that training via SFT either suffers from a distribution mismatch between the
  training data and the model's own responses or implicitly prefers only a
  certain mode of correction behavior that is often not effective at test time.
  SCoRe addresses these challenges by training under the model's own distribution
  of self-generated correction traces and using appropriate regularization to
  steer the learning process into learning a self-correction strategy that is
  effective at test time as opposed to simply fitting high-reward responses for a
  given prompt. This regularization prescribes running a first phase of RL on a
  base model to generate a policy initialization that is less susceptible to
  collapse and then using a reward bonus to amplify self-correction during
  training. When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that
  SCoRe achieves state-of-the-art self-correction performance, improving the base
  models' self-correction by 15.6% and 9.1% respectively on the MATH and
  HumanEval benchmarks.
paper_authors._id: 66ed017df63ffc857d30af8c
paper_authors.name: Aviral Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:48:30.128Z
paper_authors._id: 64879130cc0e54f272b5dbcc
paper_authors.avatarUrl: /avatars/ead1211a56a4e5c14cb8f6b2ea59c0e6.svg
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralkumar
paper_authors._id: 66ed017df63ffc857d30af8d
paper_authors.name: Vincent Zhuang
paper_authors._id: 66ed017df63ffc857d30af8e
paper_authors.name: Rishabh Agarwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:48:50.526Z
paper_authors._id: 65318c1022350ffeaf7726ef
paper_authors.avatarUrl: /avatars/9359e63d38cb69d08da2de07ae933885.svg
paper_authors.fullname: Rishabh Agarwal
paper_authors.type: user
paper_authors.user: rishabhagarwal
paper_authors._id: 66ed017df63ffc857d30af8f
paper_authors.name: Yi Su
paper_authors._id: 66ed017df63ffc857d30af90
paper_authors.name: John D Co-Reyes
paper_authors._id: 66ed017df63ffc857d30af91
paper_authors.name: Avi Singh
paper_authors._id: 66ed017df63ffc857d30af92
paper_authors.name: Kate Baumli
paper_authors._id: 66ed017df63ffc857d30af93
paper_authors.name: Shariq Iqbal
paper_authors._id: 66ed017df63ffc857d30af94
paper_authors.name: Colton Bishop
paper_authors._id: 66ed017df63ffc857d30af95
paper_authors.name: Rebecca Roelofs
paper_authors._id: 66ed017df63ffc857d30af96
paper_authors.name: Lei M Zhang
paper_authors._id: 66ed017df63ffc857d30af97
paper_authors.name: Kay McKinney
paper_authors._id: 66ed017df63ffc857d30af98
paper_authors.name: Disha Shrivastava
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:49:52.218Z
paper_authors._id: 6438284b819f3ab20d185a3b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6438284b819f3ab20d185a3b/b29SoH7_Bji6GfCszjqOE.jpeg
paper_authors.fullname: Disha Shrivastava
paper_authors.type: user
paper_authors.user: shrivasd
paper_authors._id: 66ed017df63ffc857d30af99
paper_authors.name: Cosmin Paduraru
paper_authors._id: 66ed017df63ffc857d30af9a
paper_authors.name: George Tucker
paper_authors._id: 66ed017df63ffc857d30af9b
paper_authors.name: Doina Precup
paper_authors._id: 66ed017df63ffc857d30af9c
paper_authors.name: Feryal Behbahani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:45:36.990Z
paper_authors._id: 64c197b7ec3c6181351b2c5d
paper_authors.avatarUrl: /avatars/f81fe98f68720b95885d4968b0e7cc0f.svg
paper_authors.fullname: Feryal Behbahani
paper_authors.type: user
paper_authors.user: ferya
paper_authors._id: 66ed017df63ffc857d30af9d
paper_authors.name: Aleksandra Faust
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-20T08:48:09.115Z
paper_authors._id: 64a3b828225b2121f6b3beb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a3b828225b2121f6b3beb3/44AgAQENc7cJMOWcoK4g_.jpeg
paper_authors.fullname: Aleksandra Faust
paper_authors.type: user
paper_authors.user: sandraorion
paper_title: Training Language Models to Self-Correct via Reinforcement Learning
paper_summary: Self-correction is a highly desirable capability of large language models
  (LLMs), yet it has consistently been found to be largely ineffective in modern
  LLMs. Existing approaches for training self-correction either require multiple
  models or rely on a more capable model or other forms of supervision. To this
  end, we develop a multi-turn online reinforcement learning (RL) approach,
  SCoRe, that significantly improves an LLM's self-correction ability using
  entirely self-generated data. To build SCoRe, we first show that variants of
  supervised fine-tuning (SFT) on offline model-generated correction traces are
  insufficient for instilling self-correction behavior. In particular, we observe
  that training via SFT either suffers from a distribution mismatch between the
  training data and the model's own responses or implicitly prefers only a
  certain mode of correction behavior that is often not effective at test time.
  SCoRe addresses these challenges by training under the model's own distribution
  of self-generated correction traces and using appropriate regularization to
  steer the learning process into learning a self-correction strategy that is
  effective at test time as opposed to simply fitting high-reward responses for a
  given prompt. This regularization prescribes running a first phase of RL on a
  base model to generate a policy initialization that is less susceptible to
  collapse and then using a reward bonus to amplify self-correction during
  training. When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that
  SCoRe achieves state-of-the-art self-correction performance, improving the base
  models' self-correction by 15.6% and 9.1% respectively on the MATH and
  HumanEval benchmarks.
paper_ai_summary: The multi-turn online reinforcement learning approach SCoRe improves LLM self-correction using self-generated data and regularization, achieving state-of-the-art performance.
paper_ai_keywords: reinforcement learning RL self-correction supervised fine-tuning SFT distribution mismatch reward bonus policy initialization Gemini 1.0 Pro Gemini 1.5 Flash MATH benchmark HumanEval benchmark