EAGLE-3: Scaling up Inference Acceleration of Large Language Models via
  Training-Time Test
The sequential nature of modern LLMs makes them expensive and slow, and
speculative sampling has proven to be an effective solution to this problem.
Methods like EAGLE perform autoregression at the feature level, reusing
top-layer features from the target model to achieve better results than vanilla
speculative sampling. A growing trend in the LLM community is scaling up
training data to improve model intelligence without increasing inference costs.
However, we observe that scaling up data provides limited improvements for
EAGLE. We identify that this limitation arises from EAGLE's feature prediction
constraints. In this paper, we introduce EAGLE-3, which abandons feature
prediction in favor of direct token prediction and replaces reliance on
top-layer features with multi-layer feature fusion via a technique named
training-time test. These improvements significantly enhance performance and
enable the draft model to fully benefit from scaling up training data. Our
experiments include both chat models and reasoning models, evaluated on five
tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with
about 1.4x improvement over EAGLE-2. The code is available at
https://github.com/SafeAILab/EAGLE.
67cef1d137e15ce6cd66a4fb
Yuhui Li
67cef1d137e15ce6cd66a4fc
Fangyun Wei
67cef1d137e15ce6cd66a4fd
Chao Zhang
67cef1d137e15ce6cd66a4fe
Hongyang Zhang
extracted_pending
2025-03-10T14:06:09.903Z
67ccb8fff68cfbec9dfbb658
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HrIbjkCfydS7Ne2Cmj47E.png
Hongyang Zhang
user
hongyanz
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via
  Training-Time Test
The sequential nature of modern LLMs makes them expensive and slow, and
speculative sampling has proven to be an effective solution to this problem.
Methods like EAGLE perform autoregression at the feature level, reusing
top-layer features from the target model to achieve better results than vanilla
speculative sampling. A growing trend in the LLM community is scaling up
training data to improve model intelligence without increasing inference costs.
However, we observe that scaling up data provides limited improvements for
EAGLE. We identify that this limitation arises from EAGLE's feature prediction
constraints. In this paper, we introduce EAGLE-3, which abandons feature
prediction in favor of direct token prediction and replaces reliance on
top-layer features with multi-layer feature fusion via a technique named
training-time test. These improvements significantly enhance performance and
enable the draft model to fully benefit from scaling up training data. Our
experiments include both chat models and reasoning models, evaluated on five
tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with
about 1.4x improvement over EAGLE-2. The code is available at
https://github.com/SafeAILab/EAGLE.
EAGLE-3 improves speculative sampling in large language models by replacing feature prediction with direct token prediction and using multi-layer feature fusion, leading to better performance and greater benefit from increased training data.
LLMs speculative sampling EAGLE feature level top-layer features multi-layer feature fusion training-time test token prediction