title: ReFT: Reasoning with Reinforced Fine-Tuning
summary: One way to enhance the reasoning capability of Large Language Models (LLMs)
  is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT)
  annotations. This approach does not show sufficiently strong generalization
  ability, however, because the training only relies on the given CoT data. In
  math problem-solving, for example, there is usually only one annotated
  reasoning path for each question in the training data. Intuitively, it would be
  better for the algorithm to learn from multiple annotated reasoning paths given
  a question. To address this issue, we propose a simple yet effective approach
  called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of
  learning LLMs for reasoning, with math problem-solving as an example. ReFT
  first warmups the model with SFT, and then employs on-line reinforcement
  learning, specifically the PPO algorithm in this paper, to further fine-tune
  the model, where an abundance of reasoning paths are automatically sampled
  given the question and the rewards are naturally derived from the ground-truth
  answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that
  ReFT significantly outperforms SFT, and the performance can be potentially
  further boosted by combining inference-time strategies such as majority voting
  and re-ranking. Note that ReFT obtains the improvement by learning from the
  same training questions as SFT, without relying on extra or augmented training
  questions. This indicates a superior generalization ability for ReFT.
paper_authors._id: 65a897e96cb12bdc27271f8e
paper_authors.name: Trung Quoc Luong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-18T09:20:17.737Z
paper_authors._id: 653759b362dd8126cf92a351
paper_authors.avatarUrl: /avatars/6a1e59c85e83161946f1a4fcb2d9a1d9.svg
paper_authors.fullname: Luong Quoc Trung
paper_authors.type: user
paper_authors.user: lqtrung1998
paper_authors._id: 65a897e96cb12bdc27271f8f
paper_authors.name: Xinbo Zhang
paper_authors._id: 65a897e96cb12bdc27271f90
paper_authors.name: Zhanming Jie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:10:21.390Z
paper_authors._id: 5f729da2f1e7ef6e919a1c27
paper_authors.avatarUrl: /avatars/bdf48f0d3290fab54d7e9636a3d14158.svg
paper_authors.fullname: Zhanming (Allan) Jie
paper_authors.type: user
paper_authors.user: allanjie
paper_authors._id: 65a897e96cb12bdc27271f91
paper_authors.name: Peng Sun
paper_authors._id: 65a897e96cb12bdc27271f92
paper_authors.name: Xiaoran Jin
paper_authors._id: 65a897e96cb12bdc27271f93
paper_authors.name: Hang Li
paper_title: ReFT: Reasoning with Reinforced Fine-Tuning
paper_summary: One way to enhance the reasoning capability of Large Language Models (LLMs)
  is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT)
  annotations. This approach does not show sufficiently strong generalization
  ability, however, because the training only relies on the given CoT data. In
  math problem-solving, for example, there is usually only one annotated
  reasoning path for each question in the training data. Intuitively, it would be
  better for the algorithm to learn from multiple annotated reasoning paths given
  a question. To address this issue, we propose a simple yet effective approach
  called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of
  learning LLMs for reasoning, with math problem-solving as an example. ReFT
  first warmups the model with SFT, and then employs on-line reinforcement
  learning, specifically the PPO algorithm in this paper, to further fine-tune
  the model, where an abundance of reasoning paths are automatically sampled
  given the question and the rewards are naturally derived from the ground-truth
  answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that
  ReFT significantly outperforms SFT, and the performance can be potentially
  further boosted by combining inference-time strategies such as majority voting
  and re-ranking. Note that ReFT obtains the improvement by learning from the
  same training questions as SFT, without relying on extra or augmented training
  questions. This indicates a superior generalization ability for ReFT.
paper_ai_summary: Reinforced Fine-Tuning (ReFT) improves the generalizability of large language models in reasoning tasks like math problem-solving by using reinforcement learning to learn from multiple reasoning paths.
paper_ai_keywords: Supervised Fine-Tuning Chain-of-Thought Reinforced Fine-Tuning PPO algorithm GSM8K MathQA SVAMP majority voting re-ranking