RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning
Reinforcement learning (RL) for large language models is an energy-intensive
endeavor: training can be unstable, and the policy may gradually drift away
from its pretrained weights. We present RLEP\, -- \,Reinforcement
Learning with Experience rePlay\, -- \,a two-phase framework that first
collects verified trajectories and then replays them during subsequent
training. At every update step, the policy is optimized on mini-batches that
blend newly generated rollouts with these replayed successes. By replaying
high-quality examples, RLEP steers the model away from fruitless exploration,
focuses learning on promising reasoning paths, and delivers both faster
convergence and stronger final performance. On the Qwen2.5-Math-7B base model,
RLEP reaches baseline peak accuracy with substantially fewer updates and
ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%,
on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our
code, datasets, and checkpoints are publicly available at
https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further
research.
68747204257d4f04353702de
Hongzhi Zhang
claimed_verified
2025-07-17T08:21:31.114Z
6474b290d815855e4ef59b05
/avatars/decf3cfe8a5b2203f0520cdb3d26ee40.svg
Hongzhi Zhang
user
hongzhizhang
68747204257d4f04353702df
Jia Fu
68747204257d4f04353702e0
Jingyuan Zhang
68747204257d4f04353702e1
Kai Fu
68747204257d4f04353702e2
Qi Wang
68747204257d4f04353702e3
Fuzheng Zhang
admin_assigned
2025-07-17T09:35:13.823Z
67c5945da1661d5fa6f29adb
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/lr9VmRJF0ZFECmNx5hk3x.png
Fuzheng Zhang
user
Edrex
68747204257d4f04353702e4
Guorui Zhou
admin_assigned
2025-07-17T09:34:55.007Z
67c6c570cf87e2d2ebfc81aa
https://cdn-avatars.huggingface.co/v1/production/uploads/67c6c570cf87e2d2ebfc81aa/7qAstZtIT86Uwrz3u_anv.jpeg
Guorui Zhou
user
GuoruiZhou
RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning
Reinforcement learning (RL) for large language models is an energy-intensive
endeavor: training can be unstable, and the policy may gradually drift away
from its pretrained weights. We present RLEP\, -- \,Reinforcement
Learning with Experience rePlay\, -- \,a two-phase framework that first
collects verified trajectories and then replays them during subsequent
training. At every update step, the policy is optimized on mini-batches that
blend newly generated rollouts with these replayed successes. By replaying
high-quality examples, RLEP steers the model away from fruitless exploration,
focuses learning on promising reasoning paths, and delivers both faster
convergence and stronger final performance. On the Qwen2.5-Math-7B base model,
RLEP reaches baseline peak accuracy with substantially fewer updates and
ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%,
on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our
code, datasets, and checkpoints are publicly available at
https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further
research.
RLEP, a two-phase reinforcement learning framework with experience replay, enhances large language model training by focusing on high-quality trajectories, leading to faster convergence and improved performance on math datasets.
reinforcement learning experience replay trajectories rollouts policy optimization convergence final performance Qwen2.5-Math-7B AIME-2024 AIME-2025 AMC-2023