Recurrent Drafter for Fast Speculative Decoding in Large Language Models
In this paper, we introduce an improved approach of speculative decoding
aimed at enhancing the efficiency of serving large language models. Our method
capitalizes on the strengths of two established techniques: the classic
two-model speculative decoding approach, and the more recent single-model
approach, Medusa. Drawing inspiration from Medusa, our approach adopts a
single-model strategy for speculative decoding. However, our method
distinguishes itself by employing a single, lightweight draft head with a
recurrent dependency design, akin in essence to the small, draft model uses in
classic speculative decoding, but without the complexities of the full
transformer architecture. And because of the recurrent dependency, we can use
beam search to swiftly filter out undesired candidates with the draft head. The
outcome is a method that combines the simplicity of single-model design and
avoids the need to create a data-dependent tree attention structure only for
inference in Medusa. We empirically demonstrate the effectiveness of the
proposed method on several popular open source language models, along with a
comprehensive analysis of the trade-offs involved in adopting this approach.
65f7a52009cf7381afef6e83
Aonan Zhang
admin_assigned
2024-03-18T09:24:31.165Z
65f3e68a138c6ab771434e2d
/avatars/7bfbdb1949f73b3d8f88ae2ff73900bb.svg
Aonan Zhang
user
AonanZhang
65f7a52009cf7381afef6e84
Chong Wang
65f7a52009cf7381afef6e85
Yi Wang
65f7a52009cf7381afef6e86
Xuanyu Zhang
claimed_verified
2024-03-19T08:53:34.220Z
650b7f013d41ccbfdf152016
/avatars/ae86771d80b1d4394b3529c91d7cd31e.svg
Xuanyu Zhang
user
xuanyu-zhang
65f7a52009cf7381afef6e87
Yunfei Cheng
Recurrent Drafter for Fast Speculative Decoding in Large Language Models
In this paper, we introduce an improved approach of speculative decoding
aimed at enhancing the efficiency of serving large language models. Our method
capitalizes on the strengths of two established techniques: the classic
two-model speculative decoding approach, and the more recent single-model
approach, Medusa. Drawing inspiration from Medusa, our approach adopts a
single-model strategy for speculative decoding. However, our method
distinguishes itself by employing a single, lightweight draft head with a
recurrent dependency design, akin in essence to the small, draft model uses in
classic speculative decoding, but without the complexities of the full
transformer architecture. And because of the recurrent dependency, we can use
beam search to swiftly filter out undesired candidates with the draft head. The
outcome is a method that combines the simplicity of single-model design and
avoids the need to create a data-dependent tree attention structure only for
inference in Medusa. We empirically demonstrate the effectiveness of the
proposed method on several popular open source language models, along with a
comprehensive analysis of the trade-offs involved in adopting this approach.
Improved speculative decoding uses a single lightweight draft head with recurrent dependency, combining simplicity and efficiency of single-model design while avoiding Medusa's complexity.
speculative decoding Medusa draft head recurrent dependency beam search transformer architecture