title: Reinforcement Learning for Reasoning in Large Language Models with One
    Training Example
summary: We show that reinforcement learning with verifiable reward using one training
  example (1-shot RLVR) is effective in incentivizing the math reasoning
  capabilities of large language models (LLMs). Applying RLVR to the base model
  Qwen2.5-Math-1.5B, we identify a single example that elevates model performance
  on MATH500 from 36.0% to 73.6%, and improves the average performance across six
  common mathematical reasoning benchmarks from 17.6% to 35.7%. This result
  matches the performance obtained using the 1.2k DeepScaleR subset (MATH500:
  73.6%, average: 35.9%), which includes the aforementioned example. Similar
  substantial improvements are observed across various models (Qwen2.5-Math-7B,
  Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and
  PPO), and different math examples (many of which yield approximately 30% or
  greater improvement on MATH500 when employed as a single training example). In
  addition, we identify some interesting phenomena during 1-shot RLVR, including
  cross-domain generalization, increased frequency of self-reflection, and
  sustained test performance improvement even after the training accuracy has
  saturated, a phenomenon we term post-saturation generalization. Moreover, we
  verify that the effectiveness of 1-shot RLVR primarily arises from the policy
  gradient loss, distinguishing it from the "grokking" phenomenon. We also show
  the critical role of promoting exploration (e.g., by adding entropy loss with
  an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe
  that applying entropy loss alone, without any outcome reward, significantly
  enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings
  can inspire future work on RLVR data efficiency and encourage a re-examination
  of both recent progress and the underlying mechanisms in RLVR. Our code, model,
  and data are open source at https://github.com/ypwang61/One-Shot-RLVR
paper_authors._id: 681187ddda5ce4cbd7556714
paper_authors.name: Yiping Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T09:58:59.486Z
paper_authors._id: 653586fae778506c5b38a3f1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653586fae778506c5b38a3f1/GL_RShZhAkEZmIinA5_8E.jpeg
paper_authors.fullname: Yiping Wang
paper_authors.type: user
paper_authors.user: ypwang61
paper_authors._id: 681187ddda5ce4cbd7556715
paper_authors.name: Qing Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-30T07:56:17.953Z
paper_authors._id: 673a83b99e6f1c0d81a771fc
paper_authors.avatarUrl: /avatars/f3d8e1bf7d4c36b21adee632ea12ffe0.svg
paper_authors.fullname: Qing Yang
paper_authors.type: user
paper_authors.user: hushqyang
paper_authors._id: 681187ddda5ce4cbd7556716
paper_authors.name: Zhiyuan Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T09:59:12.620Z
paper_authors._id: 64a85e23b6512b8328f9d9e2
paper_authors.avatarUrl: /avatars/4a6b35752d3f76cb03278f52b3b43426.svg
paper_authors.fullname: Zhiyuan Zeng
paper_authors.type: user
paper_authors.user: ZhiyuanZeng
paper_authors._id: 681187ddda5ce4cbd7556717
paper_authors.name: Liliang Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T09:59:18.310Z
paper_authors._id: 63815eff4761ddfa00903762
paper_authors.avatarUrl: /avatars/3419b239d42e091586f1c51b526d88e5.svg
paper_authors.fullname: Liliang Ren
paper_authors.type: user
paper_authors.user: renll
paper_authors._id: 681187ddda5ce4cbd7556718
paper_authors.name: Lucas Liu
paper_authors._id: 681187ddda5ce4cbd7556719
paper_authors.name: Baolin Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T09:59:45.735Z
paper_authors._id: 61942296d5c2ba6daa290357
paper_authors.avatarUrl: /avatars/594021cc183c4922d48b46f43772a062.svg
paper_authors.fullname: Baolin Peng
paper_authors.type: user
paper_authors.user: Baolin
paper_authors._id: 681187ddda5ce4cbd755671a
paper_authors.name: Hao Cheng
paper_authors._id: 681187ddda5ce4cbd755671b
paper_authors.name: Xuehai He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T09:59:52.344Z
paper_authors._id: 6310493158d83e8f64dc8c55
paper_authors.avatarUrl: /avatars/5f91ac4dfec0d6a5bf7bad6094f0fd0f.svg
paper_authors.fullname: Xuehai He
paper_authors.type: user
paper_authors.user: Xuehai
paper_authors._id: 681187ddda5ce4cbd755671c
paper_authors.name: Kuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T09:59:58.392Z
paper_authors._id: 633523b131a2be3938ca1016
paper_authors.avatarUrl: /avatars/06a18f80927289bb949d9f19ffdc4bda.svg
paper_authors.fullname: Kuan Wang
paper_authors.type: user
paper_authors.user: Keynes
paper_authors._id: 681187ddda5ce4cbd755671d
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:00:04.685Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 681187ddda5ce4cbd755671e
paper_authors.name: Weizhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:00:10.823Z
paper_authors._id: 64da876370446182be5b608d
paper_authors.avatarUrl: /avatars/e412fdc71404ecdf638e416846e3ebfb.svg
paper_authors.fullname: Weizhu Chen
paper_authors.type: user
paper_authors.user: chenweizhu
paper_authors._id: 681187ddda5ce4cbd755671f
paper_authors.name: Shuohang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:00:19.855Z
paper_authors._id: 6463b2247572c66a8e625a57
paper_authors.avatarUrl: /avatars/7722fb5649d42d966ce1e478946d5f8f.svg
paper_authors.fullname: Wang
paper_authors.type: user
paper_authors.user: Shuohang
paper_authors._id: 681187ddda5ce4cbd7556720
paper_authors.name: Simon Shaolei Du
paper_authors._id: 681187ddda5ce4cbd7556721
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T10:00:33.179Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_title: Reinforcement Learning for Reasoning in Large Language Models with One
    Training Example
paper_summary: We show that reinforcement learning with verifiable reward using one training
  example (1-shot RLVR) is effective in incentivizing the math reasoning
  capabilities of large language models (LLMs). Applying RLVR to the base model
  Qwen2.5-Math-1.5B, we identify a single example that elevates model performance
  on MATH500 from 36.0% to 73.6%, and improves the average performance across six
  common mathematical reasoning benchmarks from 17.6% to 35.7%. This result
  matches the performance obtained using the 1.2k DeepScaleR subset (MATH500:
  73.6%, average: 35.9%), which includes the aforementioned example. Similar
  substantial improvements are observed across various models (Qwen2.5-Math-7B,
  Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and
  PPO), and different math examples (many of which yield approximately 30% or
  greater improvement on MATH500 when employed as a single training example). In
  addition, we identify some interesting phenomena during 1-shot RLVR, including
  cross-domain generalization, increased frequency of self-reflection, and
  sustained test performance improvement even after the training accuracy has
  saturated, a phenomenon we term post-saturation generalization. Moreover, we
  verify that the effectiveness of 1-shot RLVR primarily arises from the policy
  gradient loss, distinguishing it from the "grokking" phenomenon. We also show
  the critical role of promoting exploration (e.g., by adding entropy loss with
  an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe
  that applying entropy loss alone, without any outcome reward, significantly
  enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings
  can inspire future work on RLVR data efficiency and encourage a re-examination
  of both recent progress and the underlying mechanisms in RLVR. Our code, model,
  and data are open source at https://github.com/ypwang61/One-Shot-RLVR
paper_ai_summary: Reinforcement learning with verifiable reward using one training example significantly enhances math reasoning capabilities of large language models.
paper_ai_keywords: reinforcement learning verifiable reward 1-shot RLVR large language models Qwen2.5-Math-1.5B MATH500 mathematical reasoning benchmarks Qwen2.5-Math-7B Llama3.2-3B-Instruct DeepSeek-R1-Distill-Qwen-1.5B GRPO PPO policy gradient loss entropy loss cross-domain generalization self-reflection post-saturation generalization grokking