title: Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
summary: Speculative decoding has demonstrated its effectiveness in accelerating the
  inference of large language models while maintaining a consistent sampling
  distribution. However, the conventional approach of training a separate draft
  model to achieve a satisfactory token acceptance rate can be costly. Drawing
  inspiration from early exiting, we propose a novel self-speculative decoding
  framework Kangaroo, which uses a fixed shallow sub-network as a
  self-draft model, with the remaining layers serving as the larger target model.
  We train a lightweight and efficient adapter module on top of the sub-network
  to bridge the gap between the sub-network and the full model's representation
  ability. It is noteworthy that the inference latency of the self-draft model
  may no longer be negligible compared to the large model, necessitating
  strategies to increase the token acceptance rate while minimizing the drafting
  steps of the small model. To address this challenge, we introduce an additional
  early exiting mechanism for generating draft tokens. Specifically, we halt the
  small model's subsequent prediction during the drafting phase once the
  confidence level for the current token falls below a certain threshold.
  Extensive experiments on the Spec-Bench demonstrate the effectiveness of
  Kangaroo. Under single-sequence verification, Kangaroo achieves speedups up to
  1.68times on Spec-Bench, outperforming Medusa-1 with 88.7\% fewer additional
  parameters (67M compared to 591M). The code for Kangaroo is available at
  https://github.com/Equationliu/Kangaroo.
paper_authors._id: 66306c7acde3fb5e077eafaa
paper_authors.name: Fangcheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:21:17.800Z
paper_authors._id: 64b78295479b934973e2c40e
paper_authors.avatarUrl: /avatars/9213e385964132fa50859264a838d891.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: Fangcheng2
paper_authors._id: 66306c7acde3fb5e077eafab
paper_authors.name: Yehui Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:21:23.563Z
paper_authors._id: 64d5deb154bb9eb704f83122
paper_authors.avatarUrl: /avatars/86ce09bcca903319051e2307581a43f4.svg
paper_authors.fullname: Yehui Tang
paper_authors.type: user
paper_authors.user: tangyehui
paper_authors._id: 66306c7acde3fb5e077eafac
paper_authors.name: Zhenhua Liu
paper_authors._id: 66306c7acde3fb5e077eafad
paper_authors.name: Yunsheng Ni
paper_authors._id: 66306c7acde3fb5e077eafae
paper_authors.name: Kai Han
paper_authors._id: 66306c7acde3fb5e077eafaf
paper_authors.name: Yunhe Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:22:20.752Z
paper_authors._id: 658bdf7b925aadd43304f05c
paper_authors.avatarUrl: /avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
paper_authors.fullname: Yunhe Wang
paper_authors.type: user
paper_authors.user: MightyCrane
paper_title: Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
paper_summary: Speculative decoding has demonstrated its effectiveness in accelerating the
  inference of large language models while maintaining a consistent sampling
  distribution. However, the conventional approach of training a separate draft
  model to achieve a satisfactory token acceptance rate can be costly. Drawing
  inspiration from early exiting, we propose a novel self-speculative decoding
  framework Kangaroo, which uses a fixed shallow sub-network as a
  self-draft model, with the remaining layers serving as the larger target model.
  We train a lightweight and efficient adapter module on top of the sub-network
  to bridge the gap between the sub-network and the full model's representation
  ability. It is noteworthy that the inference latency of the self-draft model
  may no longer be negligible compared to the large model, necessitating
  strategies to increase the token acceptance rate while minimizing the drafting
  steps of the small model. To address this challenge, we introduce an additional
  early exiting mechanism for generating draft tokens. Specifically, we halt the
  small model's subsequent prediction during the drafting phase once the
  confidence level for the current token falls below a certain threshold.
  Extensive experiments on the Spec-Bench demonstrate the effectiveness of
  Kangaroo. Under single-sequence verification, Kangaroo achieves speedups up to
  1.68times on Spec-Bench, outperforming Medusa-1 with 88.7\% fewer additional
  parameters (67M compared to 591M). The code for Kangaroo is available at
  https://github.com/Equationliu/Kangaroo.
paper_ai_keywords: speculative decoding large language models token acceptance rate early exiting self-speculative decoding self-draft model adapter module inference latency early exiting mechanism confidence level Spec-Bench single-sequence verification speedups