ReFT: Reasoning with Reinforced Fine-Tuning
One way to enhance the reasoning capability of Large Language Models (LLMs)
is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT)
annotations. This approach does not show sufficiently strong generalization
ability, however, because the training only relies on the given CoT data. In
math problem-solving, for example, there is usually only one annotated
reasoning path for each question in the training data. Intuitively, it would be
better for the algorithm to learn from multiple annotated reasoning paths given
a question. To address this issue, we propose a simple yet effective approach
called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of
learning LLMs for reasoning, with math problem-solving as an example. ReFT
first warmups the model with SFT, and then employs on-line reinforcement
learning, specifically the PPO algorithm in this paper, to further fine-tune
the model, where an abundance of reasoning paths are automatically sampled
given the question and the rewards are naturally derived from the ground-truth
answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that
ReFT significantly outperforms SFT, and the performance can be potentially
further boosted by combining inference-time strategies such as majority voting
and re-ranking. Note that ReFT obtains the improvement by learning from the
same training questions as SFT, without relying on extra or augmented training
questions. This indicates a superior generalization ability for ReFT.
65a897e96cb12bdc27271f8e
Trung Quoc Luong
claimed_verified
2024-01-18T09:20:17.737Z
653759b362dd8126cf92a351
/avatars/6a1e59c85e83161946f1a4fcb2d9a1d9.svg
Luong Quoc Trung
user
lqtrung1998
65a897e96cb12bdc27271f8f
Xinbo Zhang
65a897e96cb12bdc27271f90
Zhanming Jie
admin_assigned
2024-01-18T09:10:21.390Z
5f729da2f1e7ef6e919a1c27
/avatars/bdf48f0d3290fab54d7e9636a3d14158.svg
Zhanming (Allan) Jie
user
allanjie
65a897e96cb12bdc27271f91
Peng Sun
65a897e96cb12bdc27271f92
Xiaoran Jin
65a897e96cb12bdc27271f93
Hang Li
ReFT: Reasoning with Reinforced Fine-Tuning
One way to enhance the reasoning capability of Large Language Models (LLMs)
is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT)
annotations. This approach does not show sufficiently strong generalization
ability, however, because the training only relies on the given CoT data. In
math problem-solving, for example, there is usually only one annotated
reasoning path for each question in the training data. Intuitively, it would be
better for the algorithm to learn from multiple annotated reasoning paths given
a question. To address this issue, we propose a simple yet effective approach
called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of
learning LLMs for reasoning, with math problem-solving as an example. ReFT
first warmups the model with SFT, and then employs on-line reinforcement
learning, specifically the PPO algorithm in this paper, to further fine-tune
the model, where an abundance of reasoning paths are automatically sampled
given the question and the rewards are naturally derived from the ground-truth
answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that
ReFT significantly outperforms SFT, and the performance can be potentially
further boosted by combining inference-time strategies such as majority voting
and re-ranking. Note that ReFT obtains the improvement by learning from the
same training questions as SFT, without relying on extra or augmented training
questions. This indicates a superior generalization ability for ReFT.
Reinforced Fine-Tuning (ReFT) improves the generalizability of large language models in reasoning tasks like math problem-solving by using reinforcement learning to learn from multiple reasoning paths.
Supervised Fine-Tuning Chain-of-Thought Reinforced Fine-Tuning PPO algorithm GSM8K MathQA SVAMP majority voting re-ranking