Draft Model Knows When to Stop: A Self-Verification Length Policy for
  Speculative Decoding
Speculative Decoding (SD) has become an important technique in accelerating
the inference speed of large language models. Conventional SD methods employ a
fixed draft length, which ignores the token generation difficulty across tasks.
Consequently, in this paper, we address such an issue and introduce SVIP - a
difficulty-aware dynamic draft length policy for speculative decoding systems.
Based on a theoretical lower bound of draft token acceptance rate and its
inference-time approximation, SVIP adaptively determines the lengths of draft
sequences based on the entropy of each draft token distribution. Experimental
results on mainstream SD benchmarks and frameworks demonstrate the superior
performance of SVIP, achieving up to 20\% walltime speedup on SpecBench over
baseline SD methods and 60\% speedup on MT-Bench for long-form generation of up
to 8K tokens. Moreover, SVIP is totally training-free and compatible with any
existing SD methods that generate draft tokens autoregressively. Experimental
results also show that SVIP yields consistent walltime improvement on top of
GliDe & CaPE and EAGLE-2.
6747d5adfc7953b90c88935d
Ziyin Zhang
claimed_verified
2024-11-28T12:19:43.515Z
6430bdd8cd31d174a9f900fb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
Ziyin Zhang
user
Geralt-Targaryen
6747d5adfc7953b90c88935e
Jiahao Xu
admin_assigned
2024-11-28T12:20:57.716Z
660399710f1fc2f16de18072
/avatars/c22a749cc45db693c2d9ea877c7cace4.svg
Jiahao Xu
user
Jiahao004
6747d5adfc7953b90c88935f
Tian Liang
6747d5adfc7953b90c889360
Xingyu Chen
6747d5adfc7953b90c889361
Zhiwei He
admin_assigned
2024-11-28T12:21:46.167Z
638439ca834d3558a398d035
https://cdn-avatars.huggingface.co/v1/production/uploads/1669609868550-noauth.png
Zhiwei He
user
zwhe99
6747d5adfc7953b90c889362
Rui Wang
6747d5adfc7953b90c889363
Zhaopeng Tu
claimed_verified
2024-11-28T11:44:05.551Z
67485743561b1e6f9579389f
/avatars/8a4cc63bd7be388010bc329bb74582a1.svg
Zhaopeng Tu
user
zptu
Draft Model Knows When to Stop: A Self-Verification Length Policy for
  Speculative Decoding
Speculative Decoding (SD) has become an important technique in accelerating
the inference speed of large language models. Conventional SD methods employ a
fixed draft length, which ignores the token generation difficulty across tasks.
Consequently, in this paper, we address such an issue and introduce SVIP - a
difficulty-aware dynamic draft length policy for speculative decoding systems.
Based on a theoretical lower bound of draft token acceptance rate and its
inference-time approximation, SVIP adaptively determines the lengths of draft
sequences based on the entropy of each draft token distribution. Experimental
results on mainstream SD benchmarks and frameworks demonstrate the superior
performance of SVIP, achieving up to 20\% walltime speedup on SpecBench over
baseline SD methods and 60\% speedup on MT-Bench for long-form generation of up
to 8K tokens. Moreover, SVIP is totally training-free and compatible with any
existing SD methods that generate draft tokens autoregressively. Experimental
results also show that SVIP yields consistent walltime improvement on top of
GliDe & CaPE and EAGLE-2.
A difficulty-aware dynamic draft length policy for speculative decoding improves inference speed without training, yielding significant walltime reductions across benchmarks.
speculative decoding draft length entropy token generation SpecBench MT-Bench GlIDE & CaPE EAGLE-2 draft token acceptance rate