title: UFT: Unifying Supervised and Reinforcement Fine-Tuning
summary: Post-training has demonstrated its importance in enhancing the reasoning
  capabilities of large language models (LLMs). The primary post-training methods
  can be categorized into supervised fine-tuning (SFT) and reinforcement
  fine-tuning (RFT). SFT is efficient and well-suited for small language models,
  but it may lead to overfitting and limit the reasoning abilities of larger
  models. In contrast, RFT generally yields better generalization but depends
  heavily on the strength of the base model. To address the limitations of SFT
  and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm
  that unifies SFT and RFT into a single, integrated process. UFT enables the
  model to effectively explore solutions while incorporating informative
  supervision signals, bridging the gap between memorizing and thinking
  underlying existing methods. Notably, UFT outperforms both SFT and RFT in
  general, regardless of model sizes. Furthermore, we theoretically prove that
  UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for
  the first time that unified training can exponentially accelerate convergence
  on long-horizon reasoning tasks.
paper_authors._id: 68347b01021ab1eaaf09dfec
paper_authors.name: Mingyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:53:53.037Z
paper_authors._id: 67b0b9ec55810ecdb365aa7b
paper_authors.avatarUrl: /avatars/1f4e72981e05f9b79a78fae1171e2524.svg
paper_authors.fullname: Mingyang Liu
paper_authors.type: user
paper_authors.user: liumy2010
paper_authors._id: 68347b01021ab1eaaf09dfed
paper_authors.name: Gabriele Farina
paper_authors._id: 68347b01021ab1eaaf09dfee
paper_authors.name: Asuman Ozdaglar
paper_title: UFT: Unifying Supervised and Reinforcement Fine-Tuning
paper_summary: Post-training has demonstrated its importance in enhancing the reasoning
  capabilities of large language models (LLMs). The primary post-training methods
  can be categorized into supervised fine-tuning (SFT) and reinforcement
  fine-tuning (RFT). SFT is efficient and well-suited for small language models,
  but it may lead to overfitting and limit the reasoning abilities of larger
  models. In contrast, RFT generally yields better generalization but depends
  heavily on the strength of the base model. To address the limitations of SFT
  and RFT, we propose Unified Fine-Tuning (UFT), a novel post-training paradigm
  that unifies SFT and RFT into a single, integrated process. UFT enables the
  model to effectively explore solutions while incorporating informative
  supervision signals, bridging the gap between memorizing and thinking
  underlying existing methods. Notably, UFT outperforms both SFT and RFT in
  general, regardless of model sizes. Furthermore, we theoretically prove that
  UFT breaks RFT's inherent exponential sample complexity bottleneck, showing for
  the first time that unified training can exponentially accelerate convergence
  on long-horizon reasoning tasks.
paper_ai_summary: A new post-training method, Unified Fine-Tuning (UFT), improves upon supervised and reinforcement fine-tuning for large language models by combining their benefits, achieving better generalization and faster convergence.
paper_ai_keywords: supervised fine-tuning reinforcement fine-tuning Unified Fine-Tuning long-horizon reasoning tasks sample complexity bottleneck