title: EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
summary: Auto-regressive decoding makes the inference of Large Language Models (LLMs)
  time-consuming. We propose a simple framework, EAGLE (Extrapolation Algorithm
  for Greater Language-model Efficiency), for lossless acceleration. Unlike
  traditional speculative sampling methods, EAGLE operates the drafting process
  auto-regressively at the more regular (second-top-layer) feature level and
  addresses the sampling uncertainty issues in the next-feature prediction
  problems by integrating tokens from one time step ahead. The acceleration
  provided by EAGLE is lossless: it involves no fine-tuning of the target LLM,
  and the generated text maintains the same distribution as that of vanilla
  auto-regressive decoding. As of the submission of this paper, EAGLE is the
  fastest known framework within the speculative sampling family. On MT-bench,
  EAGLE is 3x faster than vanilla decoding, 2x faster than Lookahead, and 1.6x
  faster than Medusa. Using gpt-fast, EAGLE attains on average 160 tokens/s with
  LLaMA2-Chat 13B on a single RTX 3090 GPU, compared to 24 tokens/s of
  Huggingface's implementations.
paper_authors._id: 65b70e5e625ac670a7e8dcdf
paper_authors.name: Yuhui Li
paper_authors._id: 65b70e5e625ac670a7e8dce0
paper_authors.name: Fangyun Wei
paper_authors._id: 65b70e5e625ac670a7e8dce1
paper_authors.name: Chao Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-08T21:48:24.721Z
paper_authors._id: 67ccb8fff68cfbec9dfbb658
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HrIbjkCfydS7Ne2Cmj47E.png
paper_authors.fullname: Hongyang Zhang
paper_authors.type: user
paper_authors.user: hongyanz
paper_authors._id: 65b70e5e625ac670a7e8dce2
paper_authors.name: Hongyang Zhang
paper_title: EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
paper_summary: Auto-regressive decoding makes the inference of Large Language Models (LLMs)
  time-consuming. We propose a simple framework, EAGLE (Extrapolation Algorithm
  for Greater Language-model Efficiency), for lossless acceleration. Unlike
  traditional speculative sampling methods, EAGLE operates the drafting process
  auto-regressively at the more regular (second-top-layer) feature level and
  addresses the sampling uncertainty issues in the next-feature prediction
  problems by integrating tokens from one time step ahead. The acceleration
  provided by EAGLE is lossless: it involves no fine-tuning of the target LLM,
  and the generated text maintains the same distribution as that of vanilla
  auto-regressive decoding. As of the submission of this paper, EAGLE is the
  fastest known framework within the speculative sampling family. On MT-bench,
  EAGLE is 3x faster than vanilla decoding, 2x faster than Lookahead, and 1.6x
  faster than Medusa. Using gpt-fast, EAGLE attains on average 160 tokens/s with
  LLaMA2-Chat 13B on a single RTX 3090 GPU, compared to 24 tokens/s of
  Huggingface's implementations.
paper_ai_summary: EAGLE framework accelerates large language model inference by operating at a feature level with lossless performance relative to vanilla auto-regressive decoding.
paper_ai_keywords: auto-regressive decoding Large Language Models (LLMs) speculative sampling methods second-top-layer feature level lossless acceleration feature level sampling uncertainty next-feature prediction drafting process vanilla auto-regressive decoding MT-bench Lookahead Medusa gpt-fast LLaMA2-Chat 13B