Learning What Reinforcement Learning Can't: Interleaved Online
  Fine-Tuning for Hardest Questions
Recent advances in large language model (LLM) reasoning have shown that
sophisticated behaviors such as planning and self-reflection can emerge through
reinforcement learning (RL). However, despite these successes, RL in its
current form remains insufficient to induce capabilities that exceed the
limitations of the base model, as it is primarily optimized based on existing
knowledge of the model rather than facilitating the acquisition of new
information. To address this limitation, we employ supervised fine-tuning (SFT)
to learn what RL cannot, which enables the incorporation of new knowledge and
reasoning patterns by leveraging high-quality demonstration data. We analyze
the training dynamics of RL and SFT for LLM reasoning and find that RL excels
at maintaining and improving performance on questions within the model's
original capabilities, while SFT is more effective at enabling progress on
questions beyond the current scope of the model. Motivated by the complementary
strengths of RL and SFT, we introduce a novel training approach,
ReLIFT (Reinforcement Learning Interleaved
with Online Fine-Tuning). In ReLIFT, the model is primarily
trained using RL, but when it encounters challenging questions, high-quality
solutions are collected for fine-tuning, and the training process alternates
between RL and fine-tuning to enhance the model's reasoning abilities. ReLIFT
achieves an average improvement of over +5.2 points across five
competition-level benchmarks and one out-of-distribution benchmark compared to
other zero-RL models. Furthermore, we demonstrate that ReLIFT outperforms both
RL and SFT while using only 13\% of the detailed demonstration data,
highlighting its scalability. These results provide compelling evidence that
ReLIFT overcomes the fundamental limitations of RL and underscores the
significant potential.
6847dce63ec10bdd8ab4e011
Lu Ma
extracted_pending
2025-06-10T07:21:11.443Z
659e3ea885956d2cccda2b9e
/avatars/f26d415f7c1b39550af2075769f38a91.svg
马路
user
RoadQAQ
6847dce63ec10bdd8ab4e012
Hao Liang
6847dce63ec10bdd8ab4e013
Meiyi Qiang
6847dce63ec10bdd8ab4e014
Lexiang Tang
6847dce63ec10bdd8ab4e015
Xiaochen Ma
6847dce63ec10bdd8ab4e016
Zhen Hao Wong
6847dce63ec10bdd8ab4e017
Junbo Niu
6847dce63ec10bdd8ab4e018
Chengyu Shen
6847dce63ec10bdd8ab4e019
Runming He
6847dce63ec10bdd8ab4e01a
Bin Cui
6847dce63ec10bdd8ab4e01b
Wentao Zhang
Learning What Reinforcement Learning Can't: Interleaved Online
  Fine-Tuning for Hardest Questions
Recent advances in large language model (LLM) reasoning have shown that
sophisticated behaviors such as planning and self-reflection can emerge through
reinforcement learning (RL). However, despite these successes, RL in its
current form remains insufficient to induce capabilities that exceed the
limitations of the base model, as it is primarily optimized based on existing
knowledge of the model rather than facilitating the acquisition of new
information. To address this limitation, we employ supervised fine-tuning (SFT)
to learn what RL cannot, which enables the incorporation of new knowledge and
reasoning patterns by leveraging high-quality demonstration data. We analyze
the training dynamics of RL and SFT for LLM reasoning and find that RL excels
at maintaining and improving performance on questions within the model's
original capabilities, while SFT is more effective at enabling progress on
questions beyond the current scope of the model. Motivated by the complementary
strengths of RL and SFT, we introduce a novel training approach,
ReLIFT (Reinforcement Learning Interleaved
with Online Fine-Tuning). In ReLIFT, the model is primarily
trained using RL, but when it encounters challenging questions, high-quality
solutions are collected for fine-tuning, and the training process alternates
between RL and fine-tuning to enhance the model's reasoning abilities. ReLIFT
achieves an average improvement of over +5.2 points across five
competition-level benchmarks and one out-of-distribution benchmark compared to
other zero-RL models. Furthermore, we demonstrate that ReLIFT outperforms both
RL and SFT while using only 13\% of the detailed demonstration data,
highlighting its scalability. These results provide compelling evidence that
ReLIFT overcomes the fundamental limitations of RL and underscores the
significant potential.
ReLIFT, a method combining reinforcement learning and supervised fine-tuning, enhances large language model reasoning by addressing limitations of RL through interleaved training, improving performance across benchmarks with minimal data.
reinforcement learning supervised fine-tuning ReLIFT large language model reasoning training dynamics zero-RL models competition-level benchmarks out-of-distribution benchmark