title: Draft Model Knows When to Stop: A Self-Verification Length Policy for
    Speculative Decoding
summary: Speculative Decoding (SD) has become an important technique in accelerating
  the inference speed of large language models. Conventional SD methods employ a
  fixed draft length, which ignores the token generation difficulty across tasks.
  Consequently, in this paper, we address such an issue and introduce SVIP - a
  difficulty-aware dynamic draft length policy for speculative decoding systems.
  Based on a theoretical lower bound of draft token acceptance rate and its
  inference-time approximation, SVIP adaptively determines the lengths of draft
  sequences based on the entropy of each draft token distribution. Experimental
  results on mainstream SD benchmarks and frameworks demonstrate the superior
  performance of SVIP, achieving up to 20\% walltime speedup on SpecBench over
  baseline SD methods and 60\% speedup on MT-Bench for long-form generation of up
  to 8K tokens. Moreover, SVIP is totally training-free and compatible with any
  existing SD methods that generate draft tokens autoregressively. Experimental
  results also show that SVIP yields consistent walltime improvement on top of
  GliDe & CaPE and EAGLE-2.
paper_authors._id: 6747d5adfc7953b90c88935d
paper_authors.name: Ziyin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T12:19:43.515Z
paper_authors._id: 6430bdd8cd31d174a9f900fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Y9SPnRfpKSbYc7MhNdP-H.jpeg
paper_authors.fullname: Ziyin Zhang
paper_authors.type: user
paper_authors.user: Geralt-Targaryen
paper_authors._id: 6747d5adfc7953b90c88935e
paper_authors.name: Jiahao Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T12:20:57.716Z
paper_authors._id: 660399710f1fc2f16de18072
paper_authors.avatarUrl: /avatars/c22a749cc45db693c2d9ea877c7cace4.svg
paper_authors.fullname: Jiahao Xu
paper_authors.type: user
paper_authors.user: Jiahao004
paper_authors._id: 6747d5adfc7953b90c88935f
paper_authors.name: Tian Liang
paper_authors._id: 6747d5adfc7953b90c889360
paper_authors.name: Xingyu Chen
paper_authors._id: 6747d5adfc7953b90c889361
paper_authors.name: Zhiwei He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T12:21:46.167Z
paper_authors._id: 638439ca834d3558a398d035
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669609868550-noauth.png
paper_authors.fullname: Zhiwei He
paper_authors.type: user
paper_authors.user: zwhe99
paper_authors._id: 6747d5adfc7953b90c889362
paper_authors.name: Rui Wang
paper_authors._id: 6747d5adfc7953b90c889363
paper_authors.name: Zhaopeng Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T11:44:05.551Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_title: Draft Model Knows When to Stop: A Self-Verification Length Policy for
    Speculative Decoding
paper_summary: Speculative Decoding (SD) has become an important technique in accelerating
  the inference speed of large language models. Conventional SD methods employ a
  fixed draft length, which ignores the token generation difficulty across tasks.
  Consequently, in this paper, we address such an issue and introduce SVIP - a
  difficulty-aware dynamic draft length policy for speculative decoding systems.
  Based on a theoretical lower bound of draft token acceptance rate and its
  inference-time approximation, SVIP adaptively determines the lengths of draft
  sequences based on the entropy of each draft token distribution. Experimental
  results on mainstream SD benchmarks and frameworks demonstrate the superior
  performance of SVIP, achieving up to 20\% walltime speedup on SpecBench over
  baseline SD methods and 60\% speedup on MT-Bench for long-form generation of up
  to 8K tokens. Moreover, SVIP is totally training-free and compatible with any
  existing SD methods that generate draft tokens autoregressively. Experimental
  results also show that SVIP yields consistent walltime improvement on top of
  GliDe & CaPE and EAGLE-2.
paper_ai_summary: A difficulty-aware dynamic draft length policy for speculative decoding improves inference speed without training, yielding significant walltime reductions across benchmarks.
paper_ai_keywords: speculative decoding draft length entropy token generation SpecBench MT-Bench GlIDE & CaPE EAGLE-2 draft token acceptance rate