Learning to Reason under Off-Policy Guidance
Recent advances in large reasoning models (LRMs) demonstrate that
sophisticated behaviors such as multi-step reasoning and self-reflection can
emerge via reinforcement learning (RL) with simple rule-based rewards. However,
existing zero-RL approaches are inherently ``on-policy'', limiting learning to
a model's own outputs and failing to acquire reasoning abilities beyond its
initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY
guidance), a framework that augments zero-RL with off-policy reasoning traces.
LUFFY dynamically balances imitation and exploration by combining off-policy
demonstrations with on-policy rollouts during training. Notably, we propose
policy shaping via regularized importance sampling to avoid superficial and
rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an
over +7.0 average gain across six math benchmarks and an advantage of over +6.2
points in out-of-distribution tasks. It also substantially surpasses
imitation-based supervised fine-tuning (SFT), particularly in generalization.
Analysis shows LUFFY not only imitates effectively but also explores beyond
demonstrations, offering a scalable path to train generalizable reasoning
models with off-policy guidance.
6806fdeda296cac1cf860505
Jianhao Yan
admin_assigned
2025-04-22T10:43:23.332Z
6086838b19137b3a6ba760e7
/avatars/d63eea3e39b22c6e65b82c28192696f1.svg
Jianhao Yan
user
Elliott
6806fdeda296cac1cf860506
Yafu Li
admin_assigned
2025-04-22T10:43:32.087Z
63f3502a520c14618925825a
/avatars/e986a2a6625e7be6890616a417f908d2.svg
Yafu Li
user
yaful
6806fdeda296cac1cf860507
Zican Hu
admin_assigned
2025-04-22T10:43:39.458Z
662de927b10f82aa4f46557f
/avatars/19878db42ecf8e35456378571eae4643.svg
Zican Hu
user
huzican
6806fdeda296cac1cf860508
Zhi Wang
6806fdeda296cac1cf860509
Ganqu Cui
admin_assigned
2025-04-22T10:43:50.636Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
6806fdeda296cac1cf86050a
Xiaoye Qu
6806fdeda296cac1cf86050b
Yu Cheng
claimed_verified
2025-04-23T08:37:36.362Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
6806fdeda296cac1cf86050c
Yue Zhang
Learning to Reason under Off-Policy Guidance
Recent advances in large reasoning models (LRMs) demonstrate that
sophisticated behaviors such as multi-step reasoning and self-reflection can
emerge via reinforcement learning (RL) with simple rule-based rewards. However,
existing zero-RL approaches are inherently ``on-policy'', limiting learning to
a model's own outputs and failing to acquire reasoning abilities beyond its
initial capabilities. We introduce LUFFY (Learning to reason Under oFF-policY
guidance), a framework that augments zero-RL with off-policy reasoning traces.
LUFFY dynamically balances imitation and exploration by combining off-policy
demonstrations with on-policy rollouts during training. Notably, we propose
policy shaping via regularized importance sampling to avoid superficial and
rigid imitation during mixed-policy training. Remarkably, LUFFY achieves an
over +7.0 average gain across six math benchmarks and an advantage of over +6.2
points in out-of-distribution tasks. It also substantially surpasses
imitation-based supervised fine-tuning (SFT), particularly in generalization.
Analysis shows LUFFY not only imitates effectively but also explores beyond
demonstrations, offering a scalable path to train generalizable reasoning
models with off-policy guidance.
LUFFY enhances zero-RL models with off-policy guidance, improving reasoning and generalization through balanced imitation and exploration.
large reasoning models reinforcement learning zero-RL on-policy off-policy LUFFY imitation exploration policy shaping regularized importance sampling math benchmarks out-of-distribution tasks imitation-based supervised fine-tuning