title: Learning to Reason under Off-Policy Guidance
summary: Recent advances in large reasoning models (LRMs) demonstrate that
  sophisticated behaviors such as multi-step reasoning and self-reflection can
  emerge via reinforcement learning (RL) with simple rule-based rewards. However,
  existing zero-RL approaches are inherently ``on-policy'', limiting learning to
  a model's own outputs and failing to acquire reasoning abilities beyond its
  initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY
  guidance), a framework that augments zero-RL with off-policy reasoning traces.
  LUFFY dynamically balances imitation and exploration by combining off-policy
  demonstrations with on-policy rollouts during training. Notably, we propose
  policy shaping via regularized importance sampling to avoid superficial and
  rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an
  over +7.0 average gain across six math benchmarks and an advantage of over +6.2
  points in out-of-distribution tasks. It also substantially surpasses
  imitation-based supervised fine-tuning (SFT), particularly in generalization.
  Analysis shows LUFFY not only imitates effectively but also explores beyond
  demonstrations, offering a scalable path to train generalizable reasoning
  models with off-policy guidance.
paper_authors._id: 6806fdeda296cac1cf860505
paper_authors.name: Jianhao Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:43:23.332Z
paper_authors._id: 6086838b19137b3a6ba760e7
paper_authors.avatarUrl: /avatars/d63eea3e39b22c6e65b82c28192696f1.svg
paper_authors.fullname: Jianhao Yan
paper_authors.type: user
paper_authors.user: Elliott
paper_authors._id: 6806fdeda296cac1cf860506
paper_authors.name: Yafu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:43:32.087Z
paper_authors._id: 63f3502a520c14618925825a
paper_authors.avatarUrl: /avatars/e986a2a6625e7be6890616a417f908d2.svg
paper_authors.fullname: Yafu Li
paper_authors.type: user
paper_authors.user: yaful
paper_authors._id: 6806fdeda296cac1cf860507
paper_authors.name: Zican Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:43:39.458Z
paper_authors._id: 662de927b10f82aa4f46557f
paper_authors.avatarUrl: /avatars/19878db42ecf8e35456378571eae4643.svg
paper_authors.fullname: Zican Hu
paper_authors.type: user
paper_authors.user: huzican
paper_authors._id: 6806fdeda296cac1cf860508
paper_authors.name: Zhi Wang
paper_authors._id: 6806fdeda296cac1cf860509
paper_authors.name: Ganqu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-22T10:43:50.636Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 6806fdeda296cac1cf86050a
paper_authors.name: Xiaoye Qu
paper_authors._id: 6806fdeda296cac1cf86050b
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:37:36.362Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_authors._id: 6806fdeda296cac1cf86050c
paper_authors.name: Yue Zhang
paper_title: Learning to Reason under Off-Policy Guidance
paper_summary: Recent advances in large reasoning models (LRMs) demonstrate that
  sophisticated behaviors such as multi-step reasoning and self-reflection can
  emerge via reinforcement learning (RL) with simple rule-based rewards. However,
  existing zero-RL approaches are inherently ``on-policy'', limiting learning to
  a model's own outputs and failing to acquire reasoning abilities beyond its
  initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY
  guidance), a framework that augments zero-RL with off-policy reasoning traces.
  LUFFY dynamically balances imitation and exploration by combining off-policy
  demonstrations with on-policy rollouts during training. Notably, we propose
  policy shaping via regularized importance sampling to avoid superficial and
  rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an
  over +7.0 average gain across six math benchmarks and an advantage of over +6.2
  points in out-of-distribution tasks. It also substantially surpasses
  imitation-based supervised fine-tuning (SFT), particularly in generalization.
  Analysis shows LUFFY not only imitates effectively but also explores beyond
  demonstrations, offering a scalable path to train generalizable reasoning
  models with off-policy guidance.
paper_ai_summary: LUFFY enhances zero-RL models with off-policy guidance, improving reasoning and generalization through balanced imitation and exploration.
paper_ai_keywords: large reasoning models reinforcement learning zero-RL on-policy off-policy LUFFY imitation exploration policy shaping regularized importance sampling math benchmarks out-of-distribution tasks imitation-based supervised fine-tuning