## Clover-2 : Accurate Inference for Regressive Lightweight Speculative Decoding

Bin Xiao 1 Lujun Gui 2 Lei Su 1 Weipeng Chen 1

1 Baichuan Inc. 2 Beijing Institute of Technology { xiaobin, sulei, chenweipeng } @baichuan-inc.com { lujun.gui } @bit.edu.cn https://github.com/XiaoBin1992/clover

## Abstract

Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2 , an advanced iteration of Clover , an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.

## 1 Introduction

Generative Large Language Models (LLMs) [25, 1, 7], exemplified by models such as GPT, have significantly transformed the field of artificial intelligence. These models showcase exceptional adaptability, extending their applications from creative writing to engaging in human-like chatbot conversations. Their profound understanding of natural language has enhanced human-computer interactions by automating tasks that require contextual sensitivity. Nonetheless, LLMs encounter efficiency challenges when deployed on GPUs, primarily due to their sequential text generation mechanism, which involves two distinct phases: prefilling and decoding. The prefilling phase processes the entire input sequence to produce the initial token, whereas the decoding phase generates subsequent tokens iteratively, leveraging the input and previously generated tokens. The decoding phase, characterized by its repeated small-batch token processing cycles, leads to suboptimal utilization of GPU resources. This inefficiency in the decoding process represents a significant bottleneck in leveraging the full potential of these high-capacity models.

Speculative decoding [19, 9] is an acceleration technique devised to address the performance constraints associated with sequential text generation. This approach enhances computational efficiency by generating multiple tokens per step, while maintaining output consistency. The technique involves employing one or more lightweight draft models to predict several subsequent tokens with minimal computational overhead. These preliminary token predictions are then verified by the target model, allowing for the consolidation of token generations within a single iteration.

Figure 1: Overview of (a) Medusa, (b) EAGLE, (c) Clover , and (d) Clover-2 .

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00264v1-figure-b1e2f70d-4437-442c-8d7a-bf9a13fadd3c.jpg)

The effectiveness of speculative decoding is contingent on the accuracy of the initial predictions made by the draft models, which is critical for the overall decoding speed. Although more complex draft models may provide higher prediction accuracy, they can also lead to increased inference overhead and latency. Research efforts [22, 21, 23, 26, 35, 33, 15] have primarily investigated the use of independent draft models to enhance latency and throughput. In contrast, recent discussions [8, 20, 5, 4, 32, 31, 12] have illuminated the benefits of integrated speculators. These integrated approaches are noted for their lightweight architectural design and ease of deployment, offering promising directions for future advancements in speculative decoding.

Figure 1 shows the Medusa [8] solution, which uses lightweight heads for speculation. It has multiple heads that take inputs from the last transformer block's hidden states, with each layer predicting one token. To address the low hit rate from independent layer speculation, EAGLE [20] uses a target model's decoder layer as a draft model to predict tokens iteratively. It combines shifted input embeddings and the last transformer block's hidden states, reducing randomness. However, EAGLE [20] encounters challenges, primarily the suboptimal balance between speculative gains and computational expenses when employing the target model's decoder layer. For example, a 5-head sampling necessitates running the decoder layer an additional five times.

To address these challenges, we revisit our proposed Clover framework. Clover is designed specifically for real-time serving scenarios with large inference batch sizes, where traditional speculative decoding frequently encounters computational constraints, resulting in performance degradation. Clover , an RNN architecture with minimal computational requirements, has demonstrated inference speed improvements even with models exceeding 150 billion parameters and batch sizes of 48.

We introduce Clover-2 , an enhanced version of Clover . The key advancements in Clover-2 include the Information Extraction Order (Section 3.1), the Attention Decoder Output Projector (Section 3.2), and the Augmenting Block (Section 3.3). These enhancements enable speculators to leverage more sequential knowledge, thereby improving accuracy. Additionally, knowledge distillation (Section 3.4) further enhances model training performance.

Tests on Vicuan 7B and LLaMA3-Instruct 8B reveal that Clover-2 boosts throughput by up to 3.00x over standard decoding and 1.18x- 1.65x over Clover . Despite its RNN architecture, Clover-2 also delivers a maximum 7.7% speculative tokens per step and a maximum 9.3% faster speed increase on speculative heads compared to EAGLE. In summary, our key contributions are:

- · We introduce Clover-2 , an advanced version of the Clover framework, featuring upgraded model structures and the incorporation of knowledge distillation.
- · Comprehensive evaluations on Vicuan 7B and LLaMA3-Instruct 8B demonstrate that Clover-2 surpasses the efficiency of Clover and even outperforms EAGLE.

Figure 2: A demonstration of Speculative Decoding and Tree Attention. Multiple speculations are merged by prefix matching to form a tree, and its topology dependency is represented in a 2-D matrix as the casual mask in Attention computation.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00264v1-figure-c30d55a0-8a5a-4bf1-be48-93220f8b7971.jpg)

## 2 Background

## 2.1 Speculative Decoding and Tree Attention

Speculative decoding [19, 9] represents a sophisticated technique aimed at expediting the inference process of large language models (LLMs) through the enhanced utilization of hardware computational resources. This method differentiates itself from conventional auto-regressive decoding by concurrently calculating and generating multiple tokens within each iteration. At the core of speculative decoding resides a speculator component, typically a lightweight model often referred to as the draft model, tasked with predicting multiple subsequent candidate tokens (commonly structured as a tree). In the context of speculative decoding, the principal LLM, known as the target LLM, ingests all candidate tokens concurrently. This critical process is designated as the verification phase , during which the target LLM meticulously filters out any incorrect tokens from the set of speculative predictions. Consequently, speculative inference generates equivalent outputs with a reduced number of decoding steps, thereby significantly enhancing latency efficiency.

Tree Attention [22] is utilized to calculate attention scores for tree-structured candidate tokens in parallel. By applying prefix matching to various speculated sequences, the speculation results are organized into a Token Tree , which is represented as a 2-D matrix (Figure 2). It is important to note that the attention block is the only component within the modern LLM architecture that requires knowledge of sequential dependency. The scoring of tree-structured tokens is a relatively straightforward task and can be achieved by configuring the attention's Causal-Mask to align with the topological matrix. Tree Attention facilitates the integration of multiple speculations with minimal computational overhead, a feature widely implemented in many speculative decoding systems such as [14, 30, 28].

## 2.2 Clover Decoding

Clover , a lightweight speculative sampling method to address large batch sizes, introduces three incremental components to leverage sequential knowledge: Regressive Connection Attention , Decoder and Augmenting Block . The Regressive Connection enables sequential dependency from preceding speculated tokens to be considered when a speculator generates the next token. The Attention Decoder is the factual regressive block in Clover , combining the hidden states from the last transformer block and previously speculated token, merging sequential knowledge between prespeculated tokens and the entire input sentence. The Augmentation Block is an additional transformer or self-attention block appended to the target model and is used to enhance sequence features to improve speculator accuracy.

## 2.3 EAGLE Decoding

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) [20], a state-of-the-art speculative sampling method, is grounded in two key observations: first, autoregression at the feature level is simpler than at the token level, and second, feature sequences exhibit more regularity compared to token sequences. By autoregressively processing features and then deriving tokens using the LM head of the original LLM, EAGLE achieves better performance, as evidenced by a higher speedup ratio. EAGLE incorporates a single transformer decoder layer to the target LLM, ensuring easy deployment in production environments. Experimental evaluations on various models (Vicuna and LLaMA2-chat series) and tasks (multi-turn dialogue, code generation, mathematical reasoning, instruction following) demonstrate that EAGLE significantly enhances generation speed while maintaining output quality. EAGLE's innovative approach of autoregressively processing features and incorporating tokens from one time step ahead effectively mitigates sampling uncertainty, resulting in a substantial acceleration effect.

## 3 Clover-2

Figure 3: Detailed architecture design of Clover-2 .

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00264v1-figure-79783dc3-4a75-444b-b4be-9ccb9a47cc2a.jpg)

Figure 3 illustrates how Clover-2 is seamlessly integrated into existing LLMs as the speculator. Like Clover , Clover-2 incorporates three key functional modules: Regressive Connection Attention Decoder Augmenting Block . However, there are four notable differences: (1) The initial head information extraction is predefined. Clover-2 employs an independent Attention Decoder prior to the Augmenting Block to pre-integrate the hidden states and the output token information of the LLM; (2) An output projector replaces the ResBlock of Medusa with a fully connected layer, the input to this fully connected layer (FC) encompasses the hidden states and previous token embeddings; (3) Clover-2 utilizes a more sophisticated Augment Block to enhance model performance; (4) Clover-2 adopts a knowledge distillation strategy, learning not only the classification output of the LLM but also the hidden states of the LLM output.

## 3.1 Information extraction order

The Augmenting Block serves as an excellent sequence information extractor. However, in Clover, the input to the Augmenting Block lacks the information from the last token output by the LLM, potentially undermining its effectiveness. To address this, we introduced an Attention Decoder prior to the Augmenting Block to pre-summarize the hidden states and token information. Consequently, the output projection of the first head bypasses the Attention Decoder, instead directly connecting to a fully connected layer.

## 3.2 Attention Decoder output projector

In Clover-2 , the Attention Decoder output projector, previously a Medusa [8] ResBlock, is replaced with a fully connected layer. This layer accounts for both the hidden states of the Attention Decoder and the token embeddings, thereby mitigating confusion caused by the inherent uncertainty of the hidden states.

A minor adjustment to the Attention Decoder involves the removal of the SiLU activation function. Experiments have indicated that this modification does not result in performance improvements, and it is also deemed anomalous for the residual to only accumulate positive values.

The pseudocode for the Attention Decoder will be presented in Appendix A.3.

## 3.3 Augmenting Block

To condense the information from the preceding sequence into a hidden state, Clover-2 appends n additional transformer blocks following the first Attention Decoder, thereby augmenting features from the entire input sentence. Incorporating such a comprehensive layer incurs a minimal computational overhead (e.g. approximately 1 /N layer of inference time), while the accuracy gains from the augmenting block far outweigh the time it consumes. The more layers a model possesses, the smaller the proportion of computational consumption becomes.

In EAGLE [20], employing an attention decoder layer as the draft model necessitates running an additional number of head layers of attention decoder layers for each decode process. Clover-2 utilizes a lightweight Attention Decoder, with a computational load approximately 2.5 times lighter than a single layer of EAGLE, enabling the use of a more computationally intensive Augmenting Block. Such an approach is not feasible in EAGLE[20], where any additional operations incur costs that must be multiplied by the number of heads. Clover-2 adopts the simplest method, increasing the number of decoder layers in the Augmenting Block to 2.

## 3.4 Knowledge distillation

During the comparative training between Clover and Eagle[20], Clover displayed severe overfitting. Various strategies were tested without any improvement. Eventually, we observed Eagle's regression loss, which was only mentioned in the paper for auxiliary intermediate result learning. Through analysis and experimentation, we discovered that regression loss enables the draft model to focus not only on the probability of output tokens but also to more closely align with the distribution of the LLM. This represents a more profound knowledge distillation strategy, effectively suppressing overfitting and enhancing model performance. Regression loss calculates the L1 loss using the LLM's output hidden states (after normalization) and the hidden states (after normalization) output by the draft model. In Clover-2 , we refer to it as regularization loss. Consequently, our loss function was updated as follows:

$$L _ { r e g } i = S m o o t h _ { \_ } L 1 ( L L M \, h i d d e n \, s t a t e s _ { i + 1 }, \, D r a f t \, h i d d e n \, s t a t e s _ { i } ). \quad \quad ( 1 )$$

$$L _ { c l s } i = C r o s s E n t r o p y ( L L M \ p r o b _ { i + 1 }, \ D r a f t \ p r o b _ { i } ).$$

$$L = \sum _ { i = 0 } ^ { n - 1 } ( L _ { c l s } i \, + \, w _ { - } r e g \, * \, L _ { r e g } i ) \, * \, d e c a y _ { - } \, c o e f f i c i e n t ^ { i }. \quad \quad \quad$$

, where n denotes the number of draft model heads, which is 5 in Clover-2 . In the optimal practices of Clover-2 , w reg \_ is set to 10.0 and decay \_ coefficient is set to 0.7.

## 3.5 Other Details

Firstly, in Clover , a layer normalization is incorporated prior to the Attention Decoder, whereas in Clover-2 , a layer normalization is introduced before the second Attention Decoder. Additionally, a layer normalization is applied before the llm\_head, mirroring the configuration of the LLM. Similar to Clover , token embeddings are derived from the transposed matrix of the llm head weight.

Secondly, during the design process of Clover-2 , it was observed that the actual training data is not necessarily SFT data for LLM, and even SFT data exhibits distribution differences compared to

data directly decoded by LLM. To address this, we devised a sample mask strategy. Based on the token probability output by the model, we select top\_k, top\_p and compare it with the ID of the next token. If it falls within the set, the token is retained. Concurrently, different heads will be connected in series with the mask of the previous token according to the decode method. For instance, if head 1 is masked, subsequent heads will also be masked. In experiments with Llama-8B, no gains were observed. We are currently analyzing the specific reasons. Preliminary analysis suggests that the draft model learns relatively simple aspects, leaving complexity unlearned. Adding these samples is akin to introducing noise, which can prevent overfitting.

Lastly, we also designed a compressed tree mask structure, which is an additional design. This section will be included in Appendix A.1.

## 4 Evaluation

## 4.1 Experiment Settings

Models and baselines Both the EAGLE and Clover-2 approaches are employed on the Vicuna 7B v1.5 [2] and LLaMA3-Instruct 8B models [3] with the number of speculative head is 5. To ensure the fairness of the comparison, the inference engine, tree construction and tree sampling algorithm of EAGLE are used for all scenarios. We also evaluate auto-regressive decoding under the same circumstances.

Dataset and Metrics We employ the SharedGPT dataset, containing 68,000 dialogue iterations, to train both EAGLE and Clover-2 . We then evaluate inference performance on Spec-Bench[27], which includes data from MT-bench [34], WMT14 DE-EN (WMT14) [6], CNN/Daily Mail (CNN/DM) [24], Natural Questions (NQ) [18], GSM8K [11], and DPR [16], representing the tasks of multi-turn conversation, translation, summarization, question answering, mathematical reasoning, and retrieval-augmented generation, respectively. We choose extra generated tokens (i.e. tokens/step ) and throughput (i.e. tokens/second ) as our main metrics, followed by prior speculative decoding works.

Training Both models are trained with all weights frozen in the target model. For EAGLE, the initial weight settings correspond to the configuration given in [20]. While for Clover-2 , the initial weight settings correspond to the configuration given in Appendix A.2. We train Clover-2 for 20 epochs (about 4000 steps per epoch), with ( β 1 = 0 9 . , β 2 = 0 95) . for the AdamW optimizer. The learning rate 1 is set to 1e-3 with linear schedule(warmup-steps=1000, final-min-lr=5e-4).

## 4.2 End-to-end Results

Figure 4: Number of extra generated tokens (excluding the first one) per step on various tasks.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00264v1-figure-d8fc4771-716a-4668-bfe5-705ccab93200.jpg)

We evaluate the end-to-end performance at different target LLMs and tasks. Figure 4 illustrates the average number of tokens generated per step for Clover-2 and EAGLE methods on different tasks with greedy decoding. Note that the value on the vertical axis is the extra tokens per step,

1 Linear decay is applied to the learning rate.

Table 1: End-to-end throughput on Vicuan 7B v1.5 (V 7B) and LLaMA3-Instruction 8B (L 8B) with different decoding methods on six tasks. Temperature value of 0 represents greedy decoding for the target LLM, while Temperature value of 1 represents non-greedy decoding.

| Model           | Approach        | Tokens/second and Speedup rate over Vanilla Decoding   | Tokens/second and Speedup rate over Vanilla Decoding   | Tokens/second and Speedup rate over Vanilla Decoding   | Tokens/second and Speedup rate over Vanilla Decoding   | Tokens/second and Speedup rate over Vanilla Decoding   | Tokens/second and Speedup rate over Vanilla Decoding   |
|-----------------|-----------------|--------------------------------------------------------|--------------------------------------------------------|--------------------------------------------------------|--------------------------------------------------------|--------------------------------------------------------|--------------------------------------------------------|
|                 |                 | MT-bench                                               | WMT14                                                  | CNN/DM                                                 | NQ                                                     | GSM8K                                                  | DPR                                                    |
| Temperature = 0 | Temperature = 0 | Temperature = 0                                        | Temperature = 0                                        | Temperature = 0                                        | Temperature = 0                                        | Temperature = 0                                        | Temperature = 0                                        |
| V 7B            | Clover-2        | 145.6 3.00x                                            | 111.0 2.43x                                            | 121.9 2.55x                                            | 115.0 2.40x                                            | 149.2 3.09x                                            | 107.6 2.44x                                            |
| V 7B            | EAGLE           | 142.6 2.94x                                            | 106.8 2.34x                                            | 120.6 2.52x                                            | 107.4 2.24x                                            | 141.9 2.94x                                            | 102.8 2.34x                                            |
|                 | Clover-2        | 121.0 2.47x                                            | 108.6 2.24x                                            | 99.5 2.09x                                             | 103.2 2.08x                                            | 132.0 2.65x                                            | 97.4 2.14x                                             |
| L 8B            | EAGLE           | 113.6                                                  | 106.7 2.20x                                            | 96.0 2.02x = 1                                         | 100.1 2.02x                                            | 123.4 2.47x                                            | 96.5 2.12x                                             |
| L 8B            |                 | 2.32x                                                  | Temperature                                            | 98.0                                                   |                                                        |                                                        |                                                        |
| V 7B            | Clover-2        | 112.9                                                  | 90.0                                                   |                                                        | 93.2                                                   | 116.9                                                  | 87.8                                                   |
| V 7B            |                 | 2.25x 109.9                                            | 1.95x 82.3                                             | 2.03x 95.1 1.97x                                       | 1.91x 91.4 1.87x                                       | 2.39x 113.6                                            | 1.98x 87.7                                             |
| V 7B            | EAGLE           | 2.19x 98.5                                             | 1.79x 87.6                                             | 81.3 1.80x                                             | 81.6 1.77x                                             | 2.32x 105.5                                            | 1.98x 81.6                                             |
| V 7B            | Clover-2        | 2.16x                                                  | 1.93x                                                  |                                                        |                                                        | 2.28x                                                  | 1.90x                                                  |
| L 8B            |                 | 91.3                                                   |                                                        | 78.1                                                   | 78.0                                                   |                                                        |                                                        |
| V 7B            |                 |                                                        | 84.7                                                   |                                                        |                                                        | 97.9                                                   | 78.9                                                   |
| V 7B            | EAGLE           |                                                        |                                                        |                                                        |                                                        |                                                        | 1.84x                                                  |
| V 7B            |                 | 2.00x                                                  | 1.87x                                                  | 1.73x                                                  | 1.69x                                                  | 2.11x                                                  |                                                        |

excluding the actual token generated by target LLM, which more accurately reflects the performance of the speculator. With the support of the model structure performance, Clover-2 generates more tokens per step as EAGLE across all tasks. For model Vicuna 7B v1.5 , a maximum of 7.7% and an average of 2.9% improvement; for model LLaMA3-Instruction 8B , a maximum of 7.4% and an average of 3.6% improvement.

Table 1 displays the end-to-end throughput (i.e., tokens/second ) and the speedup rate relative to Vanilla Decoding. The results indicate that both methods achieve speedup across all tasks when compared to Vanilla Decoding. In the case of temperature being set to 0, Vicuna 7B v1.5 model shows a maximum improvement of 7.1% and an average improvement of 4.0%; the LLaMA3-Instruction 8B model exhibits a maximum improvement of 7.0% and an average improvement of 3.8%. When the temperature is set to 1, the Vicuna 7B v1.5 model demonstrates a maximum improvement of 9.3% and an average improvement of 3.3%, while the LLaMA3-Instruction 8B model presents a maximum improvement of 7.9% and an average improvement of 5.3%.

It should be emphasized that the above inference framework and sampling method use the same approach as Eagle. The framework is not an efficient implementation, the provided data is for reference purposes only. In theory, the more efficient the framework, the greater the benefits of Clover-2 , because Clover-2 has lower computational requirements and subsequent heads do not need to construct complex attention-related parameters.

## 4.3 Ablation Study

In the ablation study, we gradually add modules according to the experimental timeline to measure the effectiveness of each module compared to Clover . The main metric is the extra generated tokens (i.e., tokens/step ). Clover2 has an average improvement of about 30% compared to Clover, with relevant data presented in Table 2. The benefits brought by each module are as follows:

Knowledge distillation In the comparative experiment between Clover and EAGLE, severe overfitting was observed in Clover . To address this issue, we introduced a regularization loss based on knowledge distillation, which contributed to a 9% performance improvement. The main improvement comes from the later epochs, which continuously enhance the metrics.

Table 2: Ablation study on Vicuan 7B v1.5 with different decoding methods on six tasks, where T in the head means temperature.

|       | Approach   | Tokens/step and Speedup rate over Vanilla Decoding   | Tokens/step and Speedup rate over Vanilla Decoding   | Tokens/step and Speedup rate over Vanilla Decoding   | Tokens/step and Speedup rate over Vanilla Decoding   | Tokens/step and Speedup rate over Vanilla Decoding   | Tokens/step and Speedup rate over Vanilla Decoding   |
|-------|------------|------------------------------------------------------|------------------------------------------------------|------------------------------------------------------|------------------------------------------------------|------------------------------------------------------|------------------------------------------------------|
|       | Approach   | MT-bench                                             | WMT14                                                | CNN/DM                                               | NQ                                                   | GSM8K                                                | DPR                                                  |
| T = 0 | Clover-2   | 2.95                                                 | 2.51                                                 | 2.57                                                 | 2.16                                                 | 3.16                                                 | 2.70                                                 |
| T = 0 |            | 3.00x                                                | 2.43x                                                | 2.55x                                                | 2.40x                                                | 3.09x                                                | 2.44x                                                |
| T = 0 | Clover     | 2.46                                                 | 1.78                                                 | 1.48                                                 | 1.67                                                 | 2.52                                                 | 1.78                                                 |
|       |            | 2.27x                                                | 1.70x                                                | 1.54x                                                | 1.71x                                                | 2.15x                                                | 1.66x                                                |
|       | Clover-2   | 2.50                                                 | 2.15                                                 | 2.22                                                 | 1.82                                                 | 2.79                                                 | 2.24                                                 |
|       |            | 2.25x                                                | 1.95x                                                | 2.03x                                                | 1.91x                                                | 2.39x                                                | 1.98x                                                |
|       | Clover     | 2.03                                                 | 1.60                                                 | 1.33                                                 | 1.53                                                 | 2.16                                                 | 1.62                                                 |
|       |            | 1.88x                                                | 1.59x                                                | 1.44x                                                | 1.61x                                                | 1.94x                                                | 1.57x                                                |

Information extraction order Pre-setting the information aggregation of the first head allows for full utilization of the Augmenting Block's sequence extraction capabilities, effectively raising the performance ceiling of the draft model. This optimization resulted in a 7% improvement.

Attention Decoder output projector Modifying the output projector significantly improved the hit rate of the latter heads, contributing to a 5% gain.

Augmenting Block Enhancing the number of layers within the Augmenting Block effectively strengthens the sequence information aggregation capability, providing a 9% overall benefit.

## 5 Related Works

Since the introduction of speculative decoding for LLMs as outlined in [19, 9], numerous optimization techniques have been developed. The concept of tree attention, as explored in [22], has been widely implemented for the efficient verification of multiple speculations in a single step. Initial research efforts [17, 21, 23, 26, 35, 33, 15, 10] concentrated on enhancing independent draft models. In contrast, later studies [29, 14, 13] focused on draft model architectures that do not require additional training. More contemporary research has delved into the potential advantages of regressive speculators. Zhang et al. [32] employ a Multilayer Perceptron (MLP) layer as a regression block, Hydra [4] and ReDrafter [32] introduce a regressive component based on a Recurrent Neural Network (RNN), Eagle [20] incorporates a transformer decoder layer for speculation, Chimera [31] proposes the utilization of a Trigram Encoder and a Full Context Encoder as sophisticated regressive speculation mechanisms. The main difference in Clover-2 is the use of the Attention Decoder and Augment Block to capture sequential context information, followed by an RNN architecture to output multiple candidate tokens.

## 6 Conclusion

We present an upgraded version of Clover , named Clover-2 , which incorporates four enhancement points(Section 3.1, 3.2, 3.3, 3.4). In tests conducted against the original Clover and the current state-of-the-art EAGLE, Clover-2 not only significantly boosts the performance of Clover but also surpasses EAGLE in terms of hit rate and speed. Relative to Clover , Clover-2 achieves at least 19% increase in speculative tokens per step and an 18% improvement in speed. When compared to EAGLE, a state-of-the-art method, Clover-2 shows a maximum 7.7% more speculative tokens per step and a maximum 9.3% faster speed. These results demonstrate the effectiveness of the implemented improvements.

## References

- [1] ChatGPT: Optimizing Language Models for Dialogue, 2022. https://openai.com/blog/ chatgpt/ .
- [2] Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. https: //lmsys.org/blog/2023-03-30-vicuna/ .

| [3]   | Introducing Meta Llama 3: The most capable openly available LLM to date, 2024. https: //ai.meta.com/blog/meta-llama-3/ .                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|-------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [4]   | Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependent draft heads for medusa decoding, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| [5]   | Nikhil Bhendawade, Irina Belousova, Qichen Fu, Henry Mason, Mohammad Rastegari, and Mahyar Najibi. Speculative streaming: Fast llm inference without auxiliary models, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| [6]   | Ondˇej r Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, et al. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the ninth workshop on statistical machine translation , pages 12-58, 2014.                                                                                                                                                                                                                                                                                   |
| [7]   | Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems , pages 1877-1901, 2020. |
| [8]   | Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| [9]   | Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023.                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| [10]  | Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chen-Chuan Chang, and Jie Huang. Cascade speculative drafting for even faster llm inference, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| [11]  | Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.                                                                                                                                                                                                                                                                                                                                                                   |
| [12]  | Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, and Yang You. Glide with a cape: A low-hassle method to accelerate speculative decoding, 2024.                                                                                                                                                                                                                                                                                                                                                                                                    |
| [13]  | Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| [14]  | Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D. Lee, and Di He. Rest: Retrieval-based speculative decoding, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| [15]  | Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| [16]  | Vladimir Karpukhin, Barlas O˘uz, g Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 , 2020.                                                                                                                                                                                                                                                                                                                                                                                                 |
| [17]  | Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael WMahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 39236-39256. Curran Associates, Inc., 2023.                                                                                                                                                                                                                                                         |
| [18]  | Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics , 7:453-466, 2019.                                                                                                                                                                                                                                                                                              |

| [19]   | Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara En- gelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 19274-19286. PMLR, 23-29 Jul 2023.                                                                                                                                                                 |
|--------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [20]   | Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| [21]   | Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang. Online speculative decoding, 2023.                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| [22]   | Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , ASPLOS '24, page 932-949, New York, NY, USA, 2024. Association for Computing Machinery. |
| [23]   | Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling, 2023.                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| [24]   | Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023 , 2016.                                                                                                                                                                                                                                                                                                                                                                                     |
| [25]   | Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.                                                                                                                                                                                                                                                                                                                                                                                                      |
| [26]   | Benjamin Spector and Chris Re. Accelerating llm inference with staged speculative decoding, 2023.                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| [27]   | Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851 , 2024.                                                                                                                                                                                                                                                                                                                 |
| [28]   | Daliang Xu, Wangsong Yin, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, and Xuanzhe Liu. Llmcad: Fast and scalable on-device large language model inference, 2023.                                                                                                                                                                                                                                                                                                                                                                                                          |
| [29]   | Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models, 2023.                                                                                                                                                                                                                                                                                                                                                                                       |
| [30]   | Boxiang Yun, Yan Wang, Jieneng Chen, Huiyu Wang, Wei Shen, and Qingli Li. Spectr: Spectral transformer for hyperspectral pathology image segmentation, 2021.                                                                                                                                                                                                                                                                                                                                                                                                               |
| [31]   | Ziqian Zeng, Jiahong Yu, Qianshi Pang, Zihao Wang, Huiping Zhuang, Hongen Shao, and Xiaofeng Zou. Chimera: A lossless decoding method for accelerating large language models inference by fusing all tokens, 2024.                                                                                                                                                                                                                                                                                                                                                         |
| [32]   | Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, and Yunfei Cheng. Recurrent drafter for fast speculative decoding in large language models, 2024.                                                                                                                                                                                                                                                                                                                                                                                                                          |
| [33]   | Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft &verify: Lossless large language model acceleration via self-speculative decoding, 2023.                                                                                                                                                                                                                                                                                                                                                                                          |
| [34]   | Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.                                                                                                                                                                                                                                                                                                             |
| [35]   | Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation, 2024.                                                                                                                                                                                                                                                                                                                                           |

## A Appendix

## A.1 Compressed tree mask

| Ab.a   |
|--------|
| Ab.b   |
| Abb.a  |

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00264v1-figure-0d8e721c-17f9-485a-8ddf-d51079794584.jpg)

The compressed tree mask structure

Figure 5: The difference between compressed and regular tree mask structure

As shown in Figure 5, we designed a linear tree mask structure, confirming the mask relationship through numerical comparison.

## A.2 Model parameter initialization methods

Table 3: Clover-2 Model parameter initialization methods.

| Parameter                 | Init Method                                                                        |
|---------------------------|------------------------------------------------------------------------------------|
| embedding lookup          | lm_head weight Matrix transpose                                                    |
| Augmenting Block          | last decoder layer of base model                                                   |
| head 0 FC                 | eyes and uniform(b=0.01)                                                           |
| Attention Decoder 1st/2nd | q/k with eyes and uniform(b=0.01), v with uniform(b=0.01), bias with zero          |
| head 1-n FC               | eyes with uniform(b=0.01) for hidden state part, uniform(b=0.01) for embeding part |
| norm 1st/2nd              | base model norm                                                                    |

## A.3 The pseudocode of Attention Decoder

```
A.3  The pseudocode of Attention Decoder


class AttentionDecoder(nn.Module):

    def __init__(self, hidden_size, head_size, rms_norm_eps):
        super().__init__()
        self.head_size = head_size
        self
```
Bin Xiao Lujun Gui Lei Su Weipeng Chen
Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding
Large Language Models (LLMs) frequently suffer from inefficiencies, largely
attributable to the discord between the requirements of auto-regressive
decoding and the architecture of contemporary GPUs. Recently, regressive
lightweight speculative decoding has garnered attention for its notable
efficiency improvements in text generation tasks. This approach utilizes a
lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a
single transformer decoder layer, leveraging sequential information to
iteratively predict potential tokens. Specifically, RNN draft models are
computationally economical but tend to deliver lower accuracy, while attention
decoder layer models exhibit the opposite traits. This paper presents Clover-2,
an advanced iteration of Clover, an RNN-based draft model designed to achieve
comparable accuracy to that of attention decoder layer models while maintaining
minimal computational overhead. Clover-2 enhances the model architecture and
incorporates knowledge distillation to increase Clover's accuracy and improve
overall efficiency. We conducted experiments using the open-source Vicuna 7B
and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses
existing methods across various model architectures, showcasing its efficacy
and robustness.