title: Harnessing Negative Signals: Reinforcement Distillation from Teacher
    Data for LLM Reasoning
summary: Recent advances in model distillation demonstrate that data from advanced
  reasoning models (e.g., DeepSeek-R1, OpenAI's o1) can effectively transfer
  complex reasoning abilities to smaller, efficient student models. However,
  standard practices employ rejection sampling, discarding incorrect reasoning
  examples -- valuable, yet often underutilized data. This paper addresses the
  critical question: How can both positive and negative distilled reasoning
  traces be effectively leveraged to maximize LLM reasoning performance in an
  offline setting? To this end, We propose Reinforcement Distillation (REDI), a
  two-stage framework. Stage 1 learns from positive traces via Supervised
  Fine-Tuning (SFT). Stage 2 further refines the model using both positive and
  negative traces through our proposed REDI objective. This novel objective is a
  simple, reference-free loss function that outperforms established methods like
  DPO and SimPO in this distillation context. Our empirical evaluations
  demonstrate REDI's superiority over baseline Rejection Sampling SFT or SFT
  combined with DPO/SimPO on mathematical reasoning tasks. Notably, the
  Qwen-REDI-1.5B model, post-trained on just 131k positive and negative examples
  from the open Open-R1 dataset, achieves an 83.1% score on MATH-500 (pass@1).
  Its performance matches or surpasses that of DeepSeek-R1-Distill-Qwen-1.5B (a
  model post-trained on 800k proprietary data) across various mathematical
  reasoning benchmarks, establishing a new state-of-the-art for 1.5B models
  post-trained offline with openly available data.
paper_authors._id: 683d0ffbe41c42faceda19b2
paper_authors.name: Shuyao Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:40:56.229Z
paper_authors._id: 6587e5a4b2177de3967ff434
paper_authors.avatarUrl: /avatars/f2dfbc44eb2bff8d8d66d26db8539708.svg
paper_authors.fullname: Shuyao Xu
paper_authors.type: user
paper_authors.user: Tim-Xu
paper_authors._id: 683d0ffbe41c42faceda19b3
paper_authors.name: Cheng Peng
paper_authors._id: 683d0ffbe41c42faceda19b4
paper_authors.name: Jiangxuan Long
paper_authors._id: 683d0ffbe41c42faceda19b5
paper_authors.name: Weidi Xu
paper_authors._id: 683d0ffbe41c42faceda19b6
paper_authors.name: Wei Chu
paper_authors._id: 683d0ffbe41c42faceda19b7
paper_authors.name: Yuan Qi
paper_title: Harnessing Negative Signals: Reinforcement Distillation from Teacher
    Data for LLM Reasoning
paper_summary: Recent advances in model distillation demonstrate that data from advanced
  reasoning models (e.g., DeepSeek-R1, OpenAI's o1) can effectively transfer
  complex reasoning abilities to smaller, efficient student models. However,
  standard practices employ rejection sampling, discarding incorrect reasoning
  examples -- valuable, yet often underutilized data. This paper addresses the
  critical question: How can both positive and negative distilled reasoning
  traces be effectively leveraged to maximize LLM reasoning performance in an
  offline setting? To this end, We propose Reinforcement Distillation (REDI), a
  two-stage framework. Stage 1 learns from positive traces via Supervised
  Fine-Tuning (SFT). Stage 2 further refines the model using both positive and
  negative traces through our proposed REDI objective. This novel objective is a
  simple, reference-free loss function that outperforms established methods like
  DPO and SimPO in this distillation context. Our empirical evaluations
  demonstrate REDI's superiority over baseline Rejection Sampling SFT or SFT
  combined with DPO/SimPO on mathematical reasoning tasks. Notably, the
  Qwen-REDI-1.5B model, post-trained on just 131k positive and negative examples
  from the open Open-R1 dataset, achieves an 83.1% score on MATH-500 (pass@1).
  Its performance matches or surpasses that of DeepSeek-R1-Distill-Qwen-1.5B (a
  model post-trained on 800k proprietary data) across various mathematical
  reasoning benchmarks, establishing a new state-of-the-art for 1.5B models
  post-trained offline with openly available data.
paper_ai_summary: Reinforcement Distillation (REDI) leverages both positive and negative traces to enhance large language model reasoning performance offline, outperforming traditional methods and achieving state-of-the-art results with limited open data.
paper_ai_keywords: model distillation DeepSeek-R1 OpenAI's o1 Reinforcement Distillation (REDI) Supervised Fine-Tuning (SFT) REDI objective DPO SimPO mathematical reasoning tasks MATH-500 Qwen-REDI-1.5B DeepSeek-R1-Distill-Qwen-1.5B