title: Recurrent Drafter for Fast Speculative Decoding in Large Language Models
summary: In this paper, we introduce an improved approach of speculative decoding
  aimed at enhancing the efficiency of serving large language models. Our method
  capitalizes on the strengths of two established techniques: the classic
  two-model speculative decoding approach, and the more recent single-model
  approach, Medusa. Drawing inspiration from Medusa, our approach adopts a
  single-model strategy for speculative decoding. However, our method
  distinguishes itself by employing a single, lightweight draft head with a
  recurrent dependency design, akin in essence to the small, draft model uses in
  classic speculative decoding, but without the complexities of the full
  transformer architecture. And because of the recurrent dependency, we can use
  beam search to swiftly filter out undesired candidates with the draft head. The
  outcome is a method that combines the simplicity of single-model design and
  avoids the need to create a data-dependent tree attention structure only for
  inference in Medusa. We empirically demonstrate the effectiveness of the
  proposed method on several popular open source language models, along with a
  comprehensive analysis of the trade-offs involved in adopting this approach.
paper_authors._id: 65f7a52009cf7381afef6e83
paper_authors.name: Aonan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-18T09:24:31.165Z
paper_authors._id: 65f3e68a138c6ab771434e2d
paper_authors.avatarUrl: /avatars/7bfbdb1949f73b3d8f88ae2ff73900bb.svg
paper_authors.fullname: Aonan Zhang
paper_authors.type: user
paper_authors.user: AonanZhang
paper_authors._id: 65f7a52009cf7381afef6e84
paper_authors.name: Chong Wang
paper_authors._id: 65f7a52009cf7381afef6e85
paper_authors.name: Yi Wang
paper_authors._id: 65f7a52009cf7381afef6e86
paper_authors.name: Xuanyu Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-19T08:53:34.220Z
paper_authors._id: 650b7f013d41ccbfdf152016
paper_authors.avatarUrl: /avatars/ae86771d80b1d4394b3529c91d7cd31e.svg
paper_authors.fullname: Xuanyu Zhang
paper_authors.type: user
paper_authors.user: xuanyu-zhang
paper_authors._id: 65f7a52009cf7381afef6e87
paper_authors.name: Yunfei Cheng
paper_title: Recurrent Drafter for Fast Speculative Decoding in Large Language Models
paper_summary: In this paper, we introduce an improved approach of speculative decoding
  aimed at enhancing the efficiency of serving large language models. Our method
  capitalizes on the strengths of two established techniques: the classic
  two-model speculative decoding approach, and the more recent single-model
  approach, Medusa. Drawing inspiration from Medusa, our approach adopts a
  single-model strategy for speculative decoding. However, our method
  distinguishes itself by employing a single, lightweight draft head with a
  recurrent dependency design, akin in essence to the small, draft model uses in
  classic speculative decoding, but without the complexities of the full
  transformer architecture. And because of the recurrent dependency, we can use
  beam search to swiftly filter out undesired candidates with the draft head. The
  outcome is a method that combines the simplicity of single-model design and
  avoids the need to create a data-dependent tree attention structure only for
  inference in Medusa. We empirically demonstrate the effectiveness of the
  proposed method on several popular open source language models, along with a
  comprehensive analysis of the trade-offs involved in adopting this approach.
paper_ai_summary: Improved speculative decoding uses a single lightweight draft head with recurrent dependency, combining simplicity and efficiency of single-model design while avoiding Medusa's complexity.
paper_ai_keywords: speculative decoding Medusa draft head recurrent dependency beam search transformer architecture