title: Cascade Speculative Drafting for Even Faster LLM Inference
summary: Speculative decoding enhances the efficiency of large language models (LLMs)
  by leveraging a draft model to draft for a larger target model to review.
  However, drafting in speculative decoding involves slow autoregressive
  generation and generating tokens of different importance with the same time
  allocation. These two inefficiencies lead to its suboptimal performance. To
  address this issue, we introduce Cascade Speculative Drafting (CS. Drafting), a
  novel approach that employs two types of cascades. The Vertical Cascade
  eliminates autoregressive generation from neural models. The Horizontal Cascade
  constitutes efficient time allocation in drafting with its optimality supported
  by our theoretical analysis. Combining both cascades, our CS. Drafting
  algorithm has achieved up to 72 percent additional speedup over speculative
  decoding in our experiments while keeping the same output distribution.
paper_authors._id: 65812f361aa0a67bf7bf2dd9
paper_authors.name: Ziyi Chen
paper_authors._id: 65812f361aa0a67bf7bf2dda
paper_authors.name: Xiaocong Yang
paper_authors._id: 65812f361aa0a67bf7bf2ddb
paper_authors.name: Jiacheng Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:17:44.651Z
paper_authors._id: 650488e454b989666d042a49
paper_authors.avatarUrl: /avatars/3dc79c6f1a9dce872636dddd38a04670.svg
paper_authors.fullname: Jiacheng Lin
paper_authors.type: user
paper_authors.user: linjc16
paper_authors._id: 65812f361aa0a67bf7bf2ddc
paper_authors.name: Chenkai Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T10:17:51.104Z
paper_authors._id: 64ae0ab01db46d71a5da83ce
paper_authors.avatarUrl: /avatars/9583ce76207e4a62061aaaf006dcc9c3.svg
paper_authors.fullname: chenkai sun
paper_authors.type: user
paper_authors.user: we958
paper_authors._id: 65812f361aa0a67bf7bf2ddd
paper_authors.name: Jie Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-22T07:37:06.849Z
paper_authors._id: 64dd0d55304349e047edd96c
paper_authors.avatarUrl: /avatars/b5bc04180353fb9ef1474f01185050d3.svg
paper_authors.fullname: Jie Huang
paper_authors.type: user
paper_authors.user: jeffhj
paper_authors._id: 65812f361aa0a67bf7bf2dde
paper_authors.name: Kevin Chen-Chuan Chang
paper_title: Cascade Speculative Drafting for Even Faster LLM Inference
paper_summary: Speculative decoding enhances the efficiency of large language models (LLMs)
  by leveraging a draft model to draft for a larger target model to review.
  However, drafting in speculative decoding involves slow autoregressive
  generation and generating tokens of different importance with the same time
  allocation. These two inefficiencies lead to its suboptimal performance. To
  address this issue, we introduce Cascade Speculative Drafting (CS. Drafting), a
  novel approach that employs two types of cascades. The Vertical Cascade
  eliminates autoregressive generation from neural models. The Horizontal Cascade
  constitutes efficient time allocation in drafting with its optimality supported
  by our theoretical analysis. Combining both cascades, our CS. Drafting
  algorithm has achieved up to 72 percent additional speedup over speculative
  decoding in our experiments while keeping the same output distribution.
paper_ai_summary: Cascade Speculative Drafting eliminates autoregressive generation and optimizes time allocation in large language models, achieving significant speed improvements without altering output quality.
paper_ai_keywords: speculative decoding large language models draft model target model autoregressive generation Vertical Cascade Horizontal Cascade Cascade Speculative Drafting