title: Bridging Supervised Learning and Reinforcement Learning in Math
    Reasoning
summary: Reinforcement Learning (RL) has played a central role in the recent surge of
  LLMs' math abilities by enabling self-improvement through binary verifier
  signals. In contrast, Supervised Learning (SL) is rarely considered for such
  verification-driven training, largely due to its heavy reliance on reference
  answers and inability to reflect on mistakes. In this work, we challenge the
  prevailing notion that self-improvement is exclusive to RL and propose
  Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to
  reflect on their failures and improve autonomously with no external teachers.
  In online training, instead of throwing away self-generated negative answers,
  NFT constructs an implicit negative policy to model them. This implicit policy
  is parameterized with the same positive LLM we target to optimize on positive
  data, enabling direct policy optimization on all LLMs' generations. We conduct
  experiments on 7B and 32B models in math reasoning tasks. Results consistently
  show that through the additional leverage of negative feedback, NFT
  significantly improves over SL baselines like Rejection sampling Fine-Tuning,
  matching or even surpassing leading RL algorithms like GRPO and DAPO.
  Furthermore, we demonstrate that NFT and GRPO are actually equivalent in
  strict-on-policy training, even though they originate from entirely different
  theoretical foundations. Our experiments and theoretical findings bridge the
  gap between SL and RL methods in binary-feedback learning systems.
paper_authors._id: 68360e6c4e92d78afe73c338
paper_authors.name: Huayu Chen
paper_authors._id: 68360e6c4e92d78afe73c339
paper_authors.name: Kaiwen Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:58:52.235Z
paper_authors._id: 652bf7edc3cba555d5673c6e
paper_authors.avatarUrl: /avatars/78f6416c30203b30671f8423f061c657.svg
paper_authors.fullname: Kaiwen Zheng
paper_authors.type: user
paper_authors.user: worstcoder
paper_authors._id: 68360e6c4e92d78afe73c33a
paper_authors.name: Qinsheng Zhang
paper_authors._id: 68360e6c4e92d78afe73c33b
paper_authors.name: Ganqu Cui
paper_authors._id: 68360e6c4e92d78afe73c33c
paper_authors.name: Yin Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T19:14:27.483Z
paper_authors._id: 649f05367b57fab3a5b27c8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649f05367b57fab3a5b27c8b/UDJB4yqF2NmaRwCyTOfcl.jpeg
paper_authors.fullname: Yin Cui
paper_authors.type: user
paper_authors.user: richardaecn
paper_authors._id: 68360e6c4e92d78afe73c33d
paper_authors.name: Haotian Ye
paper_authors._id: 68360e6c4e92d78afe73c33e
paper_authors.name: Tsung-Yi Lin
paper_authors._id: 68360e6c4e92d78afe73c33f
paper_authors.name: Ming-Yu Liu
paper_authors._id: 68360e6c4e92d78afe73c340
paper_authors.name: Jun Zhu
paper_authors._id: 68360e6c4e92d78afe73c341
paper_authors.name: Haoxiang Wang
paper_title: Bridging Supervised Learning and Reinforcement Learning in Math
    Reasoning
paper_summary: Reinforcement Learning (RL) has played a central role in the recent surge of
  LLMs' math abilities by enabling self-improvement through binary verifier
  signals. In contrast, Supervised Learning (SL) is rarely considered for such
  verification-driven training, largely due to its heavy reliance on reference
  answers and inability to reflect on mistakes. In this work, we challenge the
  prevailing notion that self-improvement is exclusive to RL and propose
  Negative-aware Fine-Tuning (NFT) -- a supervised approach that enables LLMs to
  reflect on their failures and improve autonomously with no external teachers.
  In online training, instead of throwing away self-generated negative answers,
  NFT constructs an implicit negative policy to model them. This implicit policy
  is parameterized with the same positive LLM we target to optimize on positive
  data, enabling direct policy optimization on all LLMs' generations. We conduct
  experiments on 7B and 32B models in math reasoning tasks. Results consistently
  show that through the additional leverage of negative feedback, NFT
  significantly improves over SL baselines like Rejection sampling Fine-Tuning,
  matching or even surpassing leading RL algorithms like GRPO and DAPO.
  Furthermore, we demonstrate that NFT and GRPO are actually equivalent in
  strict-on-policy training, even though they originate from entirely different
  theoretical foundations. Our experiments and theoretical findings bridge the
  gap between SL and RL methods in binary-feedback learning systems.
paper_ai_summary: Negative-aware Fine-Tuning (NFT) enhances LLMs' math abilities using supervised learning with negative feedback, achieving performance comparable to RL methods.
paper_ai_keywords: Reinforcement Learning (RL) Supervised Learning (SL) Negative-aware Fine-Tuning (NFT) binary verifier signals negative answers implicit negative policy policy optimization math reasoning tasks Rejection sampling Fine-Tuning GRPO DAPO strict-on-policy training