title: Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
summary: Large Reasoning Models (LRMs) have shown remarkable capabilities in solving
  complex problems through reinforcement learning (RL), particularly by
  generating long reasoning traces. However, these extended outputs often exhibit
  substantial redundancy, which limits the efficiency of LRMs. In this paper, we
  investigate RL-based approaches to promote reasoning efficiency. Specifically,
  we first present a unified framework that formulates various efficient
  reasoning methods through the lens of length-based reward shaping. Building on
  this perspective, we propose a novel Length-bAsed StEp Reward shaping method
  (LASER), which employs a step function as the reward, controlled by a target
  length. LASER surpasses previous methods, achieving a superior Pareto-optimal
  balance between performance and efficiency. Next, we further extend LASER based
  on two key intuitions: (1) The reasoning behavior of the model evolves during
  training, necessitating reward specifications that are also adaptive and
  dynamic; (2) Rather than uniformly encouraging shorter or longer chains of
  thought (CoT), we posit that length-based reward shaping should be
  difficulty-aware i.e., it should penalize lengthy CoTs more for easy queries.
  This approach is expected to facilitate a combination of fast and slow
  thinking, leading to a better overall tradeoff. The resulting method is termed
  LASER-D (Dynamic and Difficulty-aware). Experiments on
  DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and
  DeepSeek-R1-Distill-Qwen-32B show that our approach significantly enhances both
  reasoning performance and response length efficiency. For instance, LASER-D and
  its variant achieve a +6.1 improvement on AIME2024 while reducing token usage
  by 63%. Further analysis reveals our RL-based compression produces more concise
  reasoning patterns with less redundant "self-reflections". Resources are at
  https://github.com/hkust-nlp/Laser.
paper_authors._id: 682e97bf9c1b77a503087f4f
paper_authors.name: Wei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:15:57.023Z
paper_authors._id: 6458af46f4d212d780bd7c68
paper_authors.avatarUrl: /avatars/832fd34bcc041b0b7b551873a459fc3c.svg
paper_authors.fullname: Wei Liu
paper_authors.type: user
paper_authors.user: PeterV09
paper_authors._id: 682e97bf9c1b77a503087f50
paper_authors.name: Ruochen Zhou
paper_authors._id: 682e97bf9c1b77a503087f51
paper_authors.name: Yiyun Deng
paper_authors._id: 682e97bf9c1b77a503087f52
paper_authors.name: Yuzhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:20.856Z
paper_authors._id: 6462def82a83863b97c0611e
paper_authors.avatarUrl: /avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
paper_authors.fullname: Yuzhen Huang
paper_authors.type: user
paper_authors.user: yuzhen17
paper_authors._id: 682e97bf9c1b77a503087f53
paper_authors.name: Junteng Liu
paper_authors._id: 682e97bf9c1b77a503087f54
paper_authors.name: Yuntian Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:15:54.170Z
paper_authors._id: 63081e15a670ed10f9d44229
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63081e15a670ed10f9d44229/w1b9uq-9774bMMgJbSPsS.jpeg
paper_authors.fullname: Yuntian Deng
paper_authors.type: user
paper_authors.user: yuntian-deng
paper_authors._id: 682e97bf9c1b77a503087f55
paper_authors.name: Yizhe Zhang
paper_authors._id: 682e97bf9c1b77a503087f56
paper_authors.name: Junxian He
paper_title: Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
paper_summary: Large Reasoning Models (LRMs) have shown remarkable capabilities in solving
  complex problems through reinforcement learning (RL), particularly by
  generating long reasoning traces. However, these extended outputs often exhibit
  substantial redundancy, which limits the efficiency of LRMs. In this paper, we
  investigate RL-based approaches to promote reasoning efficiency. Specifically,
  we first present a unified framework that formulates various efficient
  reasoning methods through the lens of length-based reward shaping. Building on
  this perspective, we propose a novel Length-bAsed StEp Reward shaping method
  (LASER), which employs a step function as the reward, controlled by a target
  length. LASER surpasses previous methods, achieving a superior Pareto-optimal
  balance between performance and efficiency. Next, we further extend LASER based
  on two key intuitions: (1) The reasoning behavior of the model evolves during
  training, necessitating reward specifications that are also adaptive and
  dynamic; (2) Rather than uniformly encouraging shorter or longer chains of
  thought (CoT), we posit that length-based reward shaping should be
  difficulty-aware i.e., it should penalize lengthy CoTs more for easy queries.
  This approach is expected to facilitate a combination of fast and slow
  thinking, leading to a better overall tradeoff. The resulting method is termed
  LASER-D (Dynamic and Difficulty-aware). Experiments on
  DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B, and
  DeepSeek-R1-Distill-Qwen-32B show that our approach significantly enhances both
  reasoning performance and response length efficiency. For instance, LASER-D and
  its variant achieve a +6.1 improvement on AIME2024 while reducing token usage
  by 63%. Further analysis reveals our RL-based compression produces more concise
  reasoning patterns with less redundant "self-reflections". Resources are at
  https://github.com/hkust-nlp/Laser.
paper_ai_summary: RL-based reward shaping methods, particularly LASER-D, enhance reasoning efficiency and performance in large reasoning models by dynamically adapting to difficulty and reducing redundancy.
paper_ai_keywords: reinforcement learning long reasoning traces length-based reward shaping LASER LASER-D reasoning behavior target length step function adaptive dynamic difficulty-aware chains of thought self-reflections