Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
Large Reasoning Models (LRMs) have shown remarkable capabilities in solving
complex problems through reinforcement learning (RL), particularly by
generating long reasoning traces. However, these extended outputs often exhibit
substantial redundancy, which limits the efficiency of LRMs. In this paper, we
investigate RL-based approaches to promote reasoning efficiency. Specifically,
we first present a unified framework that formulates various efficient
reasoning methods through the lens of length-based reward shaping. Building on
this perspective, we propose a novel Length-bAsed StEp Reward shaping method
(LASER), which employs a step function as the reward, controlled by a target
length. LASER surpasses previous methods, achieving a superior Pareto-optimal
balance between performance and efficiency. Next, we further extend LASER based
on two key intuitions: (1) The reasoning behavior of the model evolves during
training, necessitating reward specifications that are also adaptive and
dynamic; (2) Rather than uniformly encouraging shorter or longer chains of
thought (CoT), we posit that length-based reward shaping should be
difficulty-aware i.e., it should penalize lengthy CoTs more for easy queries.
This approach is expected to facilitate a combination of fast and slow
thinking, leading to a better overall tradeoff. The resulting method is termed
LASER-D (Dynamic and Difficulty-aware). Experiments on
DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and
DeepSeek-R1-Distill-Qwen-32B show that our approach significantly enhances both
reasoning performance and response length efficiency. For instance, LASER-D and
its variant achieve a +6.1 improvement on AIME2024 while reducing token usage
by 63%. Further analysis reveals our RL-based compression produces more concise
reasoning patterns with less redundant "self-reflections". Resources are at
https://github.com/hkust-nlp/Laser.
682e97bf9c1b77a503087f4f
Wei Liu
claimed_verified
2025-05-22T07:15:57.023Z
6458af46f4d212d780bd7c68
/avatars/832fd34bcc041b0b7b551873a459fc3c.svg
Wei Liu
user
PeterV09
682e97bf9c1b77a503087f50
Ruochen Zhou
682e97bf9c1b77a503087f51
Yiyun Deng
682e97bf9c1b77a503087f52
Yuzhen Huang
claimed_verified
2025-05-28T09:01:20.856Z
6462def82a83863b97c0611e
/avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
Yuzhen Huang
user
yuzhen17
682e97bf9c1b77a503087f53
Junteng Liu
682e97bf9c1b77a503087f54
Yuntian Deng
claimed_verified
2025-05-22T07:15:54.170Z
63081e15a670ed10f9d44229
https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
Yuntian Deng
user
yuntian-deng
682e97bf9c1b77a503087f55
Yizhe Zhang
682e97bf9c1b77a503087f56
Junxian He
Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
Large Reasoning Models (LRMs) have shown remarkable capabilities in solving
complex problems through reinforcement learning (RL), particularly by
generating long reasoning traces. However, these extended outputs often exhibit
substantial redundancy, which limits the efficiency of LRMs. In this paper, we
investigate RL-based approaches to promote reasoning efficiency. Specifically,
we first present a unified framework that formulates various efficient
reasoning methods through the lens of length-based reward shaping. Building on
this perspective, we propose a novel Length-bAsed StEp Reward shaping method
(LASER), which employs a step function as the reward, controlled by a target
length. LASER surpasses previous methods, achieving a superior Pareto-optimal
balance between performance and efficiency. Next, we further extend LASER based
on two key intuitions: (1) The reasoning behavior of the model evolves during
training, necessitating reward specifications that are also adaptive and
dynamic; (2) Rather than uniformly encouraging shorter or longer chains of
thought (CoT), we posit that length-based reward shaping should be
difficulty-aware i.e., it should penalize lengthy CoTs more for easy queries.
This approach is expected to facilitate a combination of fast and slow
thinking, leading to a better overall tradeoff. The resulting method is termed
LASER-D (Dynamic and Difficulty-aware). Experiments on
DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and
DeepSeek-R1-Distill-Qwen-32B show that our approach significantly enhances both
reasoning performance and response length efficiency. For instance, LASER-D and
its variant achieve a +6.1 improvement on AIME2024 while reducing token usage
by 63%. Further analysis reveals our RL-based compression produces more concise
reasoning patterns with less redundant "self-reflections". Resources are at
https://github.com/hkust-nlp/Laser.
RL-based reward shaping methods, particularly LASER-D, enhance reasoning efficiency and performance in large reasoning models by dynamically adapting to difficulty and reducing redundancy.
reinforcement learning long reasoning traces length-based reward shaping LASER LASER-D reasoning behavior target length step function adaptive dynamic difficulty-aware chains of thought self-reflections