title: POSS: Position Specialist Generates Better Draft for Speculative
    Decoding
summary: Speculative decoding accelerates Large Language Model (LLM) inference by
  using a small draft model to predict multiple tokens, and a large target model
  to verify these tokens in parallel. Recent studies leverage the hidden state of
  the target model to enhance draft model prediction accuracy. However, existing
  methods suffer from the degrading quality of draft token predictions at later
  positions, due to error accumulation in draft model generated features. In this
  paper, we propose Position Specialists (PosS), which consist of multiple
  position-specialized draft layers to generate tokens at assigned position(s).
  Position specialists greatly improve token acceptance rate at later positions
  per drafting round, as each specialist only needs to focus on handling a
  certain level of draft model feature deviation. Experiment results on
  Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets demonstrate that
  PosS effectively improves over baselines on average acceptance length and
  speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.
paper_authors._id: 6841db4b1a53c86e5eceae97
paper_authors.name: Langlin Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:41:48.349Z
paper_authors._id: 65e02d89574e5aa0e9ce3efa
paper_authors.avatarUrl: /avatars/2ab152a10b21d81fb1defc726b8e951a.svg
paper_authors.fullname: Langlin Huang
paper_authors.type: user
paper_authors.user: shrango
paper_authors._id: 6841db4b1a53c86e5eceae98
paper_authors.name: Chengsong Huang
paper_authors._id: 6841db4b1a53c86e5eceae99
paper_authors.name: Jixuan Leng
paper_authors._id: 6841db4b1a53c86e5eceae9a
paper_authors.name: Di Huang
paper_authors._id: 6841db4b1a53c86e5eceae9b
paper_authors.name: Jiaxin Huang
paper_title: POSS: Position Specialist Generates Better Draft for Speculative
    Decoding
paper_summary: Speculative decoding accelerates Large Language Model (LLM) inference by
  using a small draft model to predict multiple tokens, and a large target model
  to verify these tokens in parallel. Recent studies leverage the hidden state of
  the target model to enhance draft model prediction accuracy. However, existing
  methods suffer from the degrading quality of draft token predictions at later
  positions, due to error accumulation in draft model generated features. In this
  paper, we propose Position Specialists (PosS), which consist of multiple
  position-specialized draft layers to generate tokens at assigned position(s).
  Position specialists greatly improve token acceptance rate at later positions
  per drafting round, as each specialist only needs to focus on handling a
  certain level of draft model feature deviation. Experiment results on
  Llama-3-8B-Instruct and Llama-2-13B-chat across six datasets demonstrate that
  PosS effectively improves over baselines on average acceptance length and
  speed-up ratio. Our codebase is available at https://github.com/shrango/PosS.
paper_ai_summary: Position Specialists (PosS) enhance Large Language Model (LLM) inference by using position-specialized draft layers to improve token prediction accuracy and acceptance rates.
paper_ai_keywords: Speculative decoding LLM Large Language Model draft model target model token prediction Position Specialists PosS token acceptance rate feature deviation Llama-3-8B-Instruct Llama-2-13B-chat acceptance length speed-up ratio