title: A Minimalist Approach to LLM Reasoning: from Rejection Sampling to
    Reinforce
summary: Reinforcement learning (RL) has become a prevailing approach for fine-tuning
  large language models (LLMs) on complex reasoning tasks. Among recent methods,
  GRPO stands out for its empirical success in training models such as
  DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In
  this work, we revisit GRPO from a reinforce-like algorithm perspective and
  analyze its core components. Surprisingly, we find that a simple rejection
  sampling baseline, RAFT, which trains only on positively rewarded samples,
  yields competitive performance than GRPO and PPO. Our ablation studies reveal
  that GRPO's main advantage arises from discarding prompts with entirely
  incorrect responses, rather than from its reward normalization. Motivated by
  this insight, we propose Reinforce-Rej, a minimal extension of policy gradient
  that filters both entirely incorrect and entirely correct samples.
  Reinforce-Rej improves KL efficiency and stability, serving as a lightweight
  yet effective alternative to more complex RL algorithms. We advocate RAFT as a
  robust and interpretable baseline, and suggest that future advances should
  focus on more principled designs for incorporating negative samples, rather
  than relying on them indiscriminately. Our findings provide guidance for future
  work in reward-based LLM post-training.
paper_authors._id: 67ff2d4a86e7ad2b4bea1349
paper_authors.name: Wei Xiong
paper_authors._id: 67ff2d4a86e7ad2b4bea134a
paper_authors.name: Jiarui Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:36:03.515Z
paper_authors._id: 649369b34f0e40ee1a0ed5ba
paper_authors.avatarUrl: /avatars/50d0e77883579d5002906c8d29c26ec5.svg
paper_authors.fullname: Maxwell Yao
paper_authors.type: user
paper_authors.user: MaxwellJryao
paper_authors._id: 67ff2d4a86e7ad2b4bea134b
paper_authors.name: Yuhui Xu
paper_authors._id: 67ff2d4a86e7ad2b4bea134c
paper_authors.name: Bo Pang
paper_authors._id: 67ff2d4a86e7ad2b4bea134d
paper_authors.name: Lei Wang
paper_authors._id: 67ff2d4a86e7ad2b4bea134e
paper_authors.name: Doyen Sahoo
paper_authors._id: 67ff2d4a86e7ad2b4bea134f
paper_authors.name: Junnan Li
paper_authors._id: 67ff2d4a86e7ad2b4bea1350
paper_authors.name: Nan Jiang
paper_authors._id: 67ff2d4a86e7ad2b4bea1351
paper_authors.name: Tong Zhang
paper_authors._id: 67ff2d4a86e7ad2b4bea1352
paper_authors.name: Caiming Xiong
paper_authors._id: 67ff2d4a86e7ad2b4bea1353
paper_authors.name: Hanze Dong
paper_title: A Minimalist Approach to LLM Reasoning: from Rejection Sampling to
    Reinforce
paper_summary: Reinforcement learning (RL) has become a prevailing approach for fine-tuning
  large language models (LLMs) on complex reasoning tasks. Among recent methods,
  GRPO stands out for its empirical success in training models such as
  DeepSeek-R1, yet the sources of its effectiveness remain poorly understood. In
  this work, we revisit GRPO from a reinforce-like algorithm perspective and
  analyze its core components. Surprisingly, we find that a simple rejection
  sampling baseline, RAFT, which trains only on positively rewarded samples,
  yields competitive performance than GRPO and PPO. Our ablation studies reveal
  that GRPO's main advantage arises from discarding prompts with entirely
  incorrect responses, rather than from its reward normalization. Motivated by
  this insight, we propose Reinforce-Rej, a minimal extension of policy gradient
  that filters both entirely incorrect and entirely correct samples.
  Reinforce-Rej improves KL efficiency and stability, serving as a lightweight
  yet effective alternative to more complex RL algorithms. We advocate RAFT as a
  robust and interpretable baseline, and suggest that future advances should
  focus on more principled designs for incorporating negative samples, rather
  than relying on them indiscriminately. Our findings provide guidance for future
  work in reward-based LLM post-training.
paper_ai_summary: Reinforce-Rej, a minimal extension of policy gradient, outperforms GRPO and PPO in fine-tuning large language models by filtering out both entirely incorrect and correct samples, improving efficiency and stability.
paper_ai_keywords: reinforcement learning large language models GRPO DeepSeek-R1 RAFT reject sampling policy gradient Reinforce-Rej KL efficiency stability reward-based post-training