title: FR-Spec: Accelerating Large-Vocabulary Language Models via
    Frequency-Ranked Speculative Sampling
summary: Speculative sampling has emerged as an important technique for accelerating
  the auto-regressive generation process of large language models (LLMs) by
  utilizing a draft-then-verify mechanism to produce multiple tokens per forward
  pass. While state-of-the-art speculative sampling methods use only a single
  layer and a language modeling (LM) head as the draft model to achieve
  impressive layer compression, their efficiency gains are substantially reduced
  for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens.
  To address this, we present FR-Spec, a frequency-ranked speculative sampling
  framework that optimizes draft candidate selection through vocabulary space
  compression. By constraining the draft search to a frequency-prioritized token
  subset, our method reduces LM Head computation overhead by 75% while ensuring
  the equivalence of the final output distribution. Experiments across multiple
  datasets demonstrate an average of 1.12times speedup over the
  state-of-the-art speculative sampling method EAGLE-2.
paper_authors._id: 67bee83509a4524abf899511
paper_authors.name: Weilin Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:12:38.111Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 67bee83509a4524abf899512
paper_authors.name: Tengyu Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:01:09.673Z
paper_authors._id: 6391a84891023bed86fc703e
paper_authors.avatarUrl: /avatars/6e61d845aded1542e61f5220365e0d6f.svg
paper_authors.fullname: ThonyPan
paper_authors.type: user
paper_authors.user: ThonyPan
paper_authors._id: 67bee83509a4524abf899513
paper_authors.name: Xu Han
paper_authors._id: 67bee83509a4524abf899514
paper_authors.name: Yudi Zhang
paper_authors._id: 67bee83509a4524abf899515
paper_authors.name: Ao Sun
paper_authors._id: 67bee83509a4524abf899516
paper_authors.name: Yuxiang Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:12:36.389Z
paper_authors._id: 65394f942ddab2beff1cc967
paper_authors.avatarUrl: /avatars/bdd5b47618758f6de080a51834a17655.svg
paper_authors.fullname: Shawn Huang
paper_authors.type: user
paper_authors.user: hyx21
paper_authors._id: 67bee83509a4524abf899517
paper_authors.name: Kaihuo Zhang
paper_authors._id: 67bee83509a4524abf899518
paper_authors.name: Weilun Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:05:23.448Z
paper_authors._id: 6716194e503429f9f5fb5116
paper_authors.avatarUrl: /avatars/4707e293df6bdc14dc97a6b6f3b8932f.svg
paper_authors.fullname: zhao weilun
paper_authors.type: user
paper_authors.user: zwl96
paper_authors._id: 67bee83509a4524abf899519
paper_authors.name: Yuxuan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:05:30.760Z
paper_authors._id: 66d86ae34eb2eb8dc078f808
paper_authors.avatarUrl: /avatars/4fb926729092ce69087343ce767387c8.svg
paper_authors.fullname: Yuxuan Li
paper_authors.type: user
paper_authors.user: yuxuanli
paper_authors._id: 67bee83509a4524abf89951a
paper_authors.name: Jianyong Wang
paper_authors._id: 67bee83509a4524abf89951b
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T11:06:05.421Z
paper_authors._id: 6310a3cd531cc21f9e06de6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310a3cd531cc21f9e06de6a/aTGMx3O41lUARK9s3dAik.jpeg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: acharkq
paper_authors._id: 67bee83509a4524abf89951c
paper_authors.name: Maosong Sun
paper_title: FR-Spec: Accelerating Large-Vocabulary Language Models via
    Frequency-Ranked Speculative Sampling
paper_summary: Speculative sampling has emerged as an important technique for accelerating
  the auto-regressive generation process of large language models (LLMs) by
  utilizing a draft-then-verify mechanism to produce multiple tokens per forward
  pass. While state-of-the-art speculative sampling methods use only a single
  layer and a language modeling (LM) head as the draft model to achieve
  impressive layer compression, their efficiency gains are substantially reduced
  for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens.
  To address this, we present FR-Spec, a frequency-ranked speculative sampling
  framework that optimizes draft candidate selection through vocabulary space
  compression. By constraining the draft search to a frequency-prioritized token
  subset, our method reduces LM Head computation overhead by 75% while ensuring
  the equivalence of the final output distribution. Experiments across multiple
  datasets demonstrate an average of 1.12times speedup over the
  state-of-the-art speculative sampling method EAGLE-2.
paper_ai_keywords: speculative sampling auto-regressive generation process large language models (LLMs) draft-then-verify mechanism forward pass single layer language modeling (LM) head layer compression large-vocabulary LLMs frequency-ranked speculative sampling vocabulary space compression draft candidate selection frequency-prioritized token subset LM Head computation overhead final output distribution speedup