SRFT: A Single-Stage Method with Supervised and Reinforcement
  Fine-Tuning for Reasoning
Large language models (LLMs) have achieved remarkable progress in reasoning
tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and
Reinforcement Learning (RL) remains a fundamental challenge. Through
comprehensive analysis of token distributions, learning dynamics, and
integration mechanisms from entropy-based perspectives, we reveal key
differences between these paradigms: SFT induces coarse-grained global changes
to LLM policy distributions, while RL performs fine-grained selective
optimizations, with entropy serving as a critical indicator of training
effectiveness. Building on these observations, we propose Supervised
Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both
fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach
simultaneously applies SFT and RL to directly optimize the LLM using
demonstrations and self-exploration rollouts rather than through two-stage
sequential methods. Extensive experiments show that SRFT achieves 59.1% average
accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning
benchmarks and 10.9% on three out-of-distribution benchmarks.
685b5791d2ee4fac76521dc2
Yuqian Fu
claimed_verified
2025-06-25T08:08:54.115Z
670aa09d35918e99fe7ff6b1
/avatars/5cbea2284165191e96544bacf2bfb50f.svg
Yuqian Fu
user
Yuqian-Fu
685b5791d2ee4fac76521dc3
Tinghong Chen
685b5791d2ee4fac76521dc4
Jiajun Chai
685b5791d2ee4fac76521dc5
Xihuai Wang
685b5791d2ee4fac76521dc6
Songjun Tu
claimed_verified
2025-06-25T08:08:52.066Z
66e14f4142ceed655c731966
/avatars/d708562349913b89c8c4cf384628f82a.svg
SONGJUN TU
user
SONGJUNTU
685b5791d2ee4fac76521dc7
Guojun Yin
685b5791d2ee4fac76521dc8
Wei Lin
685b5791d2ee4fac76521dc9
Qichao Zhang
685b5791d2ee4fac76521dca
Yuanheng Zhu
685b5791d2ee4fac76521dcb
Dongbin Zhao
SRFT: A Single-Stage Method with Supervised and Reinforcement
  Fine-Tuning for Reasoning
Large language models (LLMs) have achieved remarkable progress in reasoning
tasks, yet the optimal integration of Supervised Fine-Tuning (SFT) and
Reinforcement Learning (RL) remains a fundamental challenge. Through
comprehensive analysis of token distributions, learning dynamics, and
integration mechanisms from entropy-based perspectives, we reveal key
differences between these paradigms: SFT induces coarse-grained global changes
to LLM policy distributions, while RL performs fine-grained selective
optimizations, with entropy serving as a critical indicator of training
effectiveness. Building on these observations, we propose Supervised
Reinforcement Fine-Tuning (SRFT), a single-stage method that unifies both
fine-tuning paradigms through entropy-aware weighting mechanisms. Our approach
simultaneously applies SFT and RL to directly optimize the LLM using
demonstrations and self-exploration rollouts rather than through two-stage
sequential methods. Extensive experiments show that SRFT achieves 59.1% average
accuracy, outperforming zero-RL methods by 9.0% on five mathematical reasoning
benchmarks and 10.9% on three out-of-distribution benchmarks.
 Supervised Reinforcement Fine-Tuning (SRFT) integrates Supervised Fine-Tuning and Reinforcement Learning through entropy-aware weighting to achieve high accuracy in language model optimization.
Large language models Supervised Fine-Tuning Reinforcement Learning token distributions learning dynamics entropy Supervised Reinforcement Fine-Tuning