Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
Speculative decoding has demonstrated its effectiveness in accelerating the
inference of large language models while maintaining a consistent sampling
distribution. However, the conventional approach of training a separate draft
model to achieve a satisfactory token acceptance rate can be costly. Drawing
inspiration from early exiting, we propose a novel self-speculative decoding
framework Kangaroo, which uses a fixed shallow sub-network as a
self-draft model, with the remaining layers serving as the larger target model.
We train a lightweight and efficient adapter module on top of the sub-network
to bridge the gap between the sub-network and the full model's representation
ability. It is noteworthy that the inference latency of the self-draft model
may no longer be negligible compared to the large model, necessitating
strategies to increase the token acceptance rate while minimizing the drafting
steps of the small model. To address this challenge, we introduce an additional
early exiting mechanism for generating draft tokens. Specifically, we halt the
small model's subsequent prediction during the drafting phase once the
confidence level for the current token falls below a certain threshold.
Extensive experiments on the Spec-Bench demonstrate the effectiveness of
Kangaroo. Under single-sequence verification, Kangaroo achieves speedups up to
1.68times on Spec-Bench, outperforming Medusa-1 with 88.7\% fewer additional
parameters (67M compared to 591M). The code for Kangaroo is available at
https://github.com/Equationliu/Kangaroo.
66306c7acde3fb5e077eafaa
Fangcheng Liu
admin_assigned
2024-04-30T08:21:17.800Z
64b78295479b934973e2c40e
/avatars/9213e385964132fa50859264a838d891.svg
liu
user
Fangcheng2
66306c7acde3fb5e077eafab
Yehui Tang
admin_assigned
2024-04-30T08:21:23.563Z
64d5deb154bb9eb704f83122
/avatars/86ce09bcca903319051e2307581a43f4.svg
Yehui Tang
user
tangyehui
66306c7acde3fb5e077eafac
Zhenhua Liu
66306c7acde3fb5e077eafad
Yunsheng Ni
66306c7acde3fb5e077eafae
Kai Han
66306c7acde3fb5e077eafaf
Yunhe Wang
admin_assigned
2024-04-30T08:22:20.752Z
658bdf7b925aadd43304f05c
/avatars/64d9e9dea27c376c3bc7b2a54efc2a46.svg
Yunhe Wang
user
MightyCrane
Kangaroo: Lossless Self-Speculative Decoding via Double Early Exiting
Speculative decoding has demonstrated its effectiveness in accelerating the
inference of large language models while maintaining a consistent sampling
distribution. However, the conventional approach of training a separate draft
model to achieve a satisfactory token acceptance rate can be costly. Drawing
inspiration from early exiting, we propose a novel self-speculative decoding
framework Kangaroo, which uses a fixed shallow sub-network as a
self-draft model, with the remaining layers serving as the larger target model.
We train a lightweight and efficient adapter module on top of the sub-network
to bridge the gap between the sub-network and the full model's representation
ability. It is noteworthy that the inference latency of the self-draft model
may no longer be negligible compared to the large model, necessitating
strategies to increase the token acceptance rate while minimizing the drafting
steps of the small model. To address this challenge, we introduce an additional
early exiting mechanism for generating draft tokens. Specifically, we halt the
small model's subsequent prediction during the drafting phase once the
confidence level for the current token falls below a certain threshold.
Extensive experiments on the Spec-Bench demonstrate the effectiveness of
Kangaroo. Under single-sequence verification, Kangaroo achieves speedups up to
1.68times on Spec-Bench, outperforming Medusa-1 with 88.7\% fewer additional
parameters (67M compared to 591M). The code for Kangaroo is available at
https://github.com/Equationliu/Kangaroo.
Kangaroo, a self-speculative decoding framework using a shallow sub-network as a self-draft model, accelerates large language model inference with improved token acceptance and reduced additional parameters.
speculative decoding large language models draft model self-speculative decoding self-draft model adapter module inference latency token acceptance rate early exiting mechanism confidence level Spec-Bench single-sequence verification