Clover: Regressive Lightweight Speculative Decoding with Sequential
  Knowledge
Large language models (LLMs) suffer from low efficiency as the mismatch
between the requirement of auto-regressive decoding and the design of most
contemporary GPUs. Specifically, billions to trillions of parameters must be
loaded to the GPU cache through its limited memory bandwidth for computation,
but only a small batch of tokens is actually computed. Consequently, the GPU
spends most of its time on memory transfer instead of computation. Recently,
parallel decoding, a type of speculative decoding algorithms, is becoming more
popular and has demonstrated impressive efficiency improvement in generation.
It introduces extra decoding heads to large models, enabling them to predict
multiple subsequent tokens simultaneously and verify these candidate
continuations in a single decoding step. However, this approach deviates from
the training objective of next token prediction used during pre-training,
resulting in a low hit rate for candidate tokens. In this paper, we propose a
new speculative decoding algorithm, Clover, which integrates sequential
knowledge into the parallel decoding process. This enhancement improves the hit
rate of speculators and thus boosts the overall efficiency. Clover transmits
the sequential knowledge from pre-speculated tokens via the Regressive
Connection, then employs an Attention Decoder to integrate these speculated
tokens. Additionally, Clover incorporates an Augmenting Block that modifies the
hidden states to better align with the purpose of speculative generation rather
than next token prediction. The experiment results demonstrate that Clover
outperforms the baseline by up to 91% on Baichuan-Small and 146% on
Baichuan-Large, respectively, and exceeds the performance of the previously
top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on
Baichuan-Large, respectively.
66331b80b1c6e12e1c9c2474
Bin Xiao
admin_assigned
2024-05-02T07:35:17.287Z
64f1568afcc7700c949c6154
/avatars/f23c1922f18731b77679b61aa89b309d.svg
BinXiao
user
BinXiao
66331b80b1c6e12e1c9c2475
Chunan Shi
66331b80b1c6e12e1c9c2476
Xiaonan Nie
admin_assigned
2024-05-02T07:35:30.232Z
64b6b81142134e053233c3c0
/avatars/5c7455d99a7a2648f77a531c9a71eb98.svg
Xiaonan Nie
user
codecaution
66331b80b1c6e12e1c9c2477
Fan Yang
claimed_verified
2024-05-02T08:12:07.076Z
641c45c921964f8f6d451d16
/avatars/da06cc603f8f9ee46ddb7dc72aae5bec.svg
FanYang
user
fairyang
66331b80b1c6e12e1c9c2478
Xiangwei Deng
admin_assigned
2024-05-02T07:35:52.579Z
65add7b1abf6d1ccb77b797c
/avatars/d386eb16053f65c0425e4878b0956fb2.svg
Xiangwei Deng
user
crybymyself
66331b80b1c6e12e1c9c2479
Lei Su
66331b80b1c6e12e1c9c247a
Weipeng Chen
admin_assigned
2024-05-02T07:36:05.412Z
6501587887b370a56ad2608e
/avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
Wei-Peng Chen
user
whenfra
66331b80b1c6e12e1c9c247b
Bin Cui
Clover: Regressive Lightweight Speculative Decoding with Sequential
  Knowledge
Large language models (LLMs) suffer from low efficiency as the mismatch
between the requirement of auto-regressive decoding and the design of most
contemporary GPUs. Specifically, billions to trillions of parameters must be
loaded to the GPU cache through its limited memory bandwidth for computation,
but only a small batch of tokens is actually computed. Consequently, the GPU
spends most of its time on memory transfer instead of computation. Recently,
parallel decoding, a type of speculative decoding algorithms, is becoming more
popular and has demonstrated impressive efficiency improvement in generation.
It introduces extra decoding heads to large models, enabling them to predict
multiple subsequent tokens simultaneously and verify these candidate
continuations in a single decoding step. However, this approach deviates from
the training objective of next token prediction used during pre-training,
resulting in a low hit rate for candidate tokens. In this paper, we propose a
new speculative decoding algorithm, Clover, which integrates sequential
knowledge into the parallel decoding process. This enhancement improves the hit
rate of speculators and thus boosts the overall efficiency. Clover transmits
the sequential knowledge from pre-speculated tokens via the Regressive
Connection, then employs an Attention Decoder to integrate these speculated
tokens. Additionally, Clover incorporates an Augmenting Block that modifies the
hidden states to better align with the purpose of speculative generation rather
than next token prediction. The experiment results demonstrate that Clover
outperforms the baseline by up to 91% on Baichuan-Small and 146% on
Baichuan-Large, respectively, and exceeds the performance of the previously
top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on
Baichuan-Large, respectively.
Clover, a new speculative decoding algorithm, enhances the efficiency of large language models by integrating sequential knowledge into parallel decoding, improving hit rates and outperforming previous methods.
large language models auto-regressive decoding parallel decoding speculative decoding decoding heads pre-speculated tokens Regressive Connection Attention Decoder Augmenting Block