Cascade Speculative Drafting for Even Faster LLM Inference
Speculative decoding enhances the efficiency of large language models (LLMs)
by leveraging a draft model to draft for a larger target model to review.
However, drafting in speculative decoding involves slow autoregressive
generation and generating tokens of different importance with the same time
allocation. These two inefficiencies lead to its suboptimal performance. To
address this issue, we introduce Cascade Speculative Drafting (CS. Drafting), a
novel approach that employs two types of cascades. The Vertical Cascade
eliminates autoregressive generation from neural models. The Horizontal Cascade
constitutes efficient time allocation in drafting with its optimality supported
by our theoretical analysis. Combining both cascades, our CS. Drafting
algorithm has achieved up to 72 percent additional speedup over speculative
decoding in our experiments while keeping the same output distribution.
65812f361aa0a67bf7bf2dd9
Ziyi Chen
65812f361aa0a67bf7bf2dda
Xiaocong Yang
65812f361aa0a67bf7bf2ddb
Jiacheng Lin
admin_assigned
2023-12-19T10:17:44.651Z
650488e454b989666d042a49
/avatars/3dc79c6f1a9dce872636dddd38a04670.svg
Jiacheng Lin
user
linjc16
65812f361aa0a67bf7bf2ddc
Chenkai Sun
admin_assigned
2023-12-19T10:17:51.104Z
64ae0ab01db46d71a5da83ce
/avatars/9583ce76207e4a62061aaaf006dcc9c3.svg
chenkai sun
user
we958
65812f361aa0a67bf7bf2ddd
Jie Huang
claimed_verified
2023-12-22T07:37:06.849Z
64dd0d55304349e047edd96c
/avatars/b5bc04180353fb9ef1474f01185050d3.svg
Jie Huang
user
jeffhj
65812f361aa0a67bf7bf2dde
Kevin Chen-Chuan Chang
Cascade Speculative Drafting for Even Faster LLM Inference
Speculative decoding enhances the efficiency of large language models (LLMs)
by leveraging a draft model to draft for a larger target model to review.
However, drafting in speculative decoding involves slow autoregressive
generation and generating tokens of different importance with the same time
allocation. These two inefficiencies lead to its suboptimal performance. To
address this issue, we introduce Cascade Speculative Drafting (CS. Drafting), a
novel approach that employs two types of cascades. The Vertical Cascade
eliminates autoregressive generation from neural models. The Horizontal Cascade
constitutes efficient time allocation in drafting with its optimality supported
by our theoretical analysis. Combining both cascades, our CS. Drafting
algorithm has achieved up to 72 percent additional speedup over speculative
decoding in our experiments while keeping the same output distribution.
Cascade Speculative Drafting eliminates autoregressive generation and optimizes time allocation in large language models, achieving significant speed improvements without altering output quality.
speculative decoding large language models draft model target model autoregressive generation Vertical Cascade Horizontal Cascade Cascade Speculative Drafting