title: Clover: Regressive Lightweight Speculative Decoding with Sequential
    Knowledge
summary: Large language models (LLMs) suffer from low efficiency as the mismatch
  between the requirement of auto-regressive decoding and the design of most
  contemporary GPUs. Specifically, billions to trillions of parameters must be
  loaded to the GPU cache through its limited memory bandwidth for computation,
  but only a small batch of tokens is actually computed. Consequently, the GPU
  spends most of its time on memory transfer instead of computation. Recently,
  parallel decoding, a type of speculative decoding algorithms, is becoming more
  popular and has demonstrated impressive efficiency improvement in generation.
  It introduces extra decoding heads to large models, enabling them to predict
  multiple subsequent tokens simultaneously and verify these candidate
  continuations in a single decoding step. However, this approach deviates from
  the training objective of next token prediction used during pre-training,
  resulting in a low hit rate for candidate tokens. In this paper, we propose a
  new speculative decoding algorithm, Clover, which integrates sequential
  knowledge into the parallel decoding process. This enhancement improves the hit
  rate of speculators and thus boosts the overall efficiency. Clover transmits
  the sequential knowledge from pre-speculated tokens via the Regressive
  Connection, then employs an Attention Decoder to integrate these speculated
  tokens. Additionally, Clover incorporates an Augmenting Block that modifies the
  hidden states to better align with the purpose of speculative generation rather
  than next token prediction. The experiment results demonstrate that Clover
  outperforms the baseline by up to 91% on Baichuan-Small and 146% on
  Baichuan-Large, respectively, and exceeds the performance of the previously
  top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on
  Baichuan-Large, respectively.
paper_authors._id: 66331b80b1c6e12e1c9c2474
paper_authors.name: Bin Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:35:17.287Z
paper_authors._id: 64f1568afcc7700c949c6154
paper_authors.avatarUrl: /avatars/f23c1922f18731b77679b61aa89b309d.svg
paper_authors.fullname: BinXiao
paper_authors.type: user
paper_authors.user: BinXiao
paper_authors._id: 66331b80b1c6e12e1c9c2475
paper_authors.name: Chunan Shi
paper_authors._id: 66331b80b1c6e12e1c9c2476
paper_authors.name: Xiaonan Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:35:30.232Z
paper_authors._id: 64b6b81142134e053233c3c0
paper_authors.avatarUrl: /avatars/5c7455d99a7a2648f77a531c9a71eb98.svg
paper_authors.fullname: Xiaonan Nie
paper_authors.type: user
paper_authors.user: codecaution
paper_authors._id: 66331b80b1c6e12e1c9c2477
paper_authors.name: Fan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T08:12:07.076Z
paper_authors._id: 641c45c921964f8f6d451d16
paper_authors.avatarUrl: /avatars/da06cc603f8f9ee46ddb7dc72aae5bec.svg
paper_authors.fullname: FanYang
paper_authors.type: user
paper_authors.user: fairyang
paper_authors._id: 66331b80b1c6e12e1c9c2478
paper_authors.name: Xiangwei Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:35:52.579Z
paper_authors._id: 65add7b1abf6d1ccb77b797c
paper_authors.avatarUrl: /avatars/d386eb16053f65c0425e4878b0956fb2.svg
paper_authors.fullname: Xiangwei Deng
paper_authors.type: user
paper_authors.user: crybymyself
paper_authors._id: 66331b80b1c6e12e1c9c2479
paper_authors.name: Lei Su
paper_authors._id: 66331b80b1c6e12e1c9c247a
paper_authors.name: Weipeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:36:05.412Z
paper_authors._id: 6501587887b370a56ad2608e
paper_authors.avatarUrl: /avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
paper_authors.fullname: Wei-Peng Chen
paper_authors.type: user
paper_authors.user: whenfra
paper_authors._id: 66331b80b1c6e12e1c9c247b
paper_authors.name: Bin Cui
paper_title: Clover: Regressive Lightweight Speculative Decoding with Sequential
    Knowledge
paper_summary: Large language models (LLMs) suffer from low efficiency as the mismatch
  between the requirement of auto-regressive decoding and the design of most
  contemporary GPUs. Specifically, billions to trillions of parameters must be
  loaded to the GPU cache through its limited memory bandwidth for computation,
  but only a small batch of tokens is actually computed. Consequently, the GPU
  spends most of its time on memory transfer instead of computation. Recently,
  parallel decoding, a type of speculative decoding algorithms, is becoming more
  popular and has demonstrated impressive efficiency improvement in generation.
  It introduces extra decoding heads to large models, enabling them to predict
  multiple subsequent tokens simultaneously and verify these candidate
  continuations in a single decoding step. However, this approach deviates from
  the training objective of next token prediction used during pre-training,
  resulting in a low hit rate for candidate tokens. In this paper, we propose a
  new speculative decoding algorithm, Clover, which integrates sequential
  knowledge into the parallel decoding process. This enhancement improves the hit
  rate of speculators and thus boosts the overall efficiency. Clover transmits
  the sequential knowledge from pre-speculated tokens via the Regressive
  Connection, then employs an Attention Decoder to integrate these speculated
  tokens. Additionally, Clover incorporates an Augmenting Block that modifies the
  hidden states to better align with the purpose of speculative generation rather
  than next token prediction. The experiment results demonstrate that Clover
  outperforms the baseline by up to 91% on Baichuan-Small and 146% on
  Baichuan-Large, respectively, and exceeds the performance of the previously
  top-performing method, Medusa, by up to 37% on Baichuan-Small and 57% on
  Baichuan-Large, respectively.
paper_ai_keywords: curved decoding speculative decoding algorithm Clover parallel decoding Regressive Connection Attention Decoder Augmenting Block pre-training Baichuan-Small Baichuan-Large Medusa