title: Reward-Guided Speculative Decoding for Efficient LLM Reasoning
summary: We introduce Reward-Guided Speculative Decoding (RSD), a novel framework
  aimed at improving the efficiency of inference in large language models (LLMs).
  RSD synergistically combines a lightweight draft model with a more powerful
  target model, incorporating a controlled bias to prioritize high-reward
  outputs, in contrast to existing speculative decoding methods that enforce
  strict unbiasedness. RSD employs a process reward model to evaluate
  intermediate decoding steps and dynamically decide whether to invoke the target
  model, optimizing the trade-off between computational cost and output quality.
  We theoretically demonstrate that a threshold-based mixture strategy achieves
  an optimal balance between resource utilization and performance. Extensive
  evaluations on challenging reasoning benchmarks, including Olympiad-level
  tasks, show that RSD delivers significant efficiency gains against decoding
  with the target model only (up to 4.4x fewer FLOPs), while achieving
  significant better accuracy than parallel decoding method on average (up to
  +3.5). These results highlight RSD as a robust and cost-effective approach for
  deploying LLMs in resource-intensive scenarios.
paper_authors._id: 67a04151dd7b3a4aba880589
paper_authors.name: Baohao Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T11:10:48.379Z
paper_authors._id: 62c414354ce7250560a1f67f
paper_authors.avatarUrl: /avatars/28fd73973d1703c84f4f59644fef8a80.svg
paper_authors.fullname: Baohao Liao
paper_authors.type: user
paper_authors.user: baohao
paper_authors._id: 67a04151dd7b3a4aba88058a
paper_authors.name: Yuhui Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-03T11:07:52.570Z
paper_authors._id: 6602869253a0518b2a98cafd
paper_authors.avatarUrl: /avatars/c14b5953a716f42c83ad28147f8308ae.svg
paper_authors.fullname: Yuhui Xu
paper_authors.type: user
paper_authors.user: yuhuixu
paper_authors._id: 67a04151dd7b3a4aba88058b
paper_authors.name: Hanze Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T11:10:54.880Z
paper_authors._id: 63a3ff69f91ad3ea5703841d
paper_authors.avatarUrl: /avatars/69227c4bce01d33747c1377b6f9672db.svg
paper_authors.fullname: Hanze Dong
paper_authors.type: user
paper_authors.user: hendrydong
paper_authors._id: 67a04151dd7b3a4aba88058c
paper_authors.name: Junnan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T11:11:08.170Z
paper_authors._id: 61f9d3b54ac99e8a1bae85f4
paper_authors.avatarUrl: /avatars/ac47d13204dd22452e4bc46e280842d5.svg
paper_authors.fullname: JunnanLi
paper_authors.type: user
paper_authors.user: JunnanLi
paper_authors._id: 67a04151dd7b3a4aba88058d
paper_authors.name: Christof Monz
paper_authors._id: 67a04151dd7b3a4aba88058e
paper_authors.name: Silvio Savarese
paper_authors._id: 67a04151dd7b3a4aba88058f
paper_authors.name: Doyen Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T11:11:23.345Z
paper_authors._id: 65f84fd980481173afd91233
paper_authors.avatarUrl: /avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
paper_authors.fullname: Doyen
paper_authors.type: user
paper_authors.user: doyensahoo
paper_authors._id: 67a04151dd7b3a4aba880590
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-03T11:11:29.821Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_title: Reward-Guided Speculative Decoding for Efficient LLM Reasoning
paper_summary: We introduce Reward-Guided Speculative Decoding (RSD), a novel framework
  aimed at improving the efficiency of inference in large language models (LLMs).
  RSD synergistically combines a lightweight draft model with a more powerful
  target model, incorporating a controlled bias to prioritize high-reward
  outputs, in contrast to existing speculative decoding methods that enforce
  strict unbiasedness. RSD employs a process reward model to evaluate
  intermediate decoding steps and dynamically decide whether to invoke the target
  model, optimizing the trade-off between computational cost and output quality.
  We theoretically demonstrate that a threshold-based mixture strategy achieves
  an optimal balance between resource utilization and performance. Extensive
  evaluations on challenging reasoning benchmarks, including Olympiad-level
  tasks, show that RSD delivers significant efficiency gains against decoding
  with the target model only (up to 4.4x fewer FLOPs), while achieving
  significant better accuracy than parallel decoding method on average (up to
  +3.5). These results highlight RSD as a robust and cost-effective approach for
  deploying LLMs in resource-intensive scenarios.
paper_ai_summary: Reward-Guided Speculative Decoding improves inference efficiency in large language models by combining a draft and target model with a reward-based strategy, achieving better performance and cost savings.
paper_ai_keywords: Reward-Guided Speculative Decoding speculative decoding draft model target model process reward model threshold-based mixture strategy computational cost output quality FLOPs parallel decoding