title: EAGLE-3: Scaling up Inference Acceleration of Large Language Models via
    Training-Time Test
summary: The sequential nature of modern LLMs makes them expensive and slow, and
  speculative sampling has proven to be an effective solution to this problem.
  Methods like EAGLE perform autoregression at the feature level, reusing
  top-layer features from the target model to achieve better results than vanilla
  speculative sampling. A growing trend in the LLM community is scaling up
  training data to improve model intelligence without increasing inference costs.
  However, we observe that scaling up data provides limited improvements for
  EAGLE. We identify that this limitation arises from EAGLE's feature prediction
  constraints. In this paper, we introduce EAGLE-3, which abandons feature
  prediction in favor of direct token prediction and replaces reliance on
  top-layer features with multi-layer feature fusion via a technique named
  training-time test. These improvements significantly enhance performance and
  enable the draft model to fully benefit from scaling up training data. Our
  experiments include both chat models and reasoning models, evaluated on five
  tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with
  about 1.4x improvement over EAGLE-2. The code is available at
  https://github.com/SafeAILab/EAGLE.
paper_authors._id: 67cef1d137e15ce6cd66a4fb
paper_authors.name: Yuhui Li
paper_authors._id: 67cef1d137e15ce6cd66a4fc
paper_authors.name: Fangyun Wei
paper_authors._id: 67cef1d137e15ce6cd66a4fd
paper_authors.name: Chao Zhang
paper_authors._id: 67cef1d137e15ce6cd66a4fe
paper_authors.name: Hongyang Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-10T14:06:09.903Z
paper_authors._id: 67ccb8fff68cfbec9dfbb658
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HrIbjkCfydS7Ne2Cmj47E.png
paper_authors.fullname: Hongyang Zhang
paper_authors.type: user
paper_authors.user: hongyanz
paper_title: EAGLE-3: Scaling up Inference Acceleration of Large Language Models via
    Training-Time Test
paper_summary: The sequential nature of modern LLMs makes them expensive and slow, and
  speculative sampling has proven to be an effective solution to this problem.
  Methods like EAGLE perform autoregression at the feature level, reusing
  top-layer features from the target model to achieve better results than vanilla
  speculative sampling. A growing trend in the LLM community is scaling up
  training data to improve model intelligence without increasing inference costs.
  However, we observe that scaling up data provides limited improvements for
  EAGLE. We identify that this limitation arises from EAGLE's feature prediction
  constraints. In this paper, we introduce EAGLE-3, which abandons feature
  prediction in favor of direct token prediction and replaces reliance on
  top-layer features with multi-layer feature fusion via a technique named
  training-time test. These improvements significantly enhance performance and
  enable the draft model to fully benefit from scaling up training data. Our
  experiments include both chat models and reasoning models, evaluated on five
  tasks. The results show that EAGLE-3 achieves a speedup ratio up to 6.5x, with
  about 1.4x improvement over EAGLE-2. The code is available at
  https://github.com/SafeAILab/EAGLE.
paper_ai_keywords: LLMs speculative sampling EAGLE autoregression feature level top-layer features multi-layer feature fusion training-time test draft model chat models reasoning models