title: DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with
    Dynamic Multi-Sequence Drafting
summary: Large language models (LLMs) exhibit exceptional performance across a wide
  range of tasks; however, their token-by-token autoregressive generation process
  significantly hinders inference speed. Speculative decoding presents a
  promising draft-then-verify framework that reduces generation latency while
  maintaining output distribution fidelity. Nevertheless, the draft model
  introduces additional computational overhead, becoming a performance bottleneck
  and increasing the time to first token (TTFT). Previous approaches to mitigate
  draft model overhead have primarily relied on heuristics and generally failed
  to match the quality of the draft language models. To address these challenges,
  we propose DuoDecoding, a novel approach that strategically deploys the draft
  and target models on the CPU and GPU respectively, enabling parallel decoding
  while preserving draft quality. Our method incorporates a hardware-aware
  optimal draft budget to minimize idle times and employs dynamic multi-sequence
  drafting to enhance draft quality. Extensive experiments across seven tasks
  show that DuoDecoding achieves up to 2.61x speedup in generation latency, while
  reducing TTFT to 83% of that in conventional speculative decoding. The Code is
  available at https://github.com/KaiLv69/DuoDecoding.
paper_authors._id: 67c673bcf47209364f0cec96
paper_authors.name: Kai Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:14:11.523Z
paper_authors._id: 6485d5b300c9cfe5c2470c81
paper_authors.avatarUrl: /avatars/c29aa81d2add795e8448b99274a04b83.svg
paper_authors.fullname: Kai
paper_authors.type: user
paper_authors.user: KaiLv
paper_authors._id: 67c673bcf47209364f0cec97
paper_authors.name: Honglin Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:14:04.672Z
paper_authors._id: 638ef0b0c67af472d31674a6
paper_authors.avatarUrl: /avatars/02df97d15a0f46b47f9162221733b121.svg
paper_authors.fullname: Honglin Guo
paper_authors.type: user
paper_authors.user: KYLN24
paper_authors._id: 67c673bcf47209364f0cec98
paper_authors.name: Qipeng Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:13:46.322Z
paper_authors._id: 6491cd52b1e5d3444528edb1
paper_authors.avatarUrl: /avatars/a85635d886c7f157b6723dec5c01c030.svg
paper_authors.fullname: Qipeng Guo
paper_authors.type: user
paper_authors.user: QipengGuo
paper_authors._id: 67c673bcf47209364f0cec99
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:13:40.885Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_title: DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with
    Dynamic Multi-Sequence Drafting
paper_summary: Large language models (LLMs) exhibit exceptional performance across a wide
  range of tasks; however, their token-by-token autoregressive generation process
  significantly hinders inference speed. Speculative decoding presents a
  promising draft-then-verify framework that reduces generation latency while
  maintaining output distribution fidelity. Nevertheless, the draft model
  introduces additional computational overhead, becoming a performance bottleneck
  and increasing the time to first token (TTFT). Previous approaches to mitigate
  draft model overhead have primarily relied on heuristics and generally failed
  to match the quality of the draft language models. To address these challenges,
  we propose DuoDecoding, a novel approach that strategically deploys the draft
  and target models on the CPU and GPU respectively, enabling parallel decoding
  while preserving draft quality. Our method incorporates a hardware-aware
  optimal draft budget to minimize idle times and employs dynamic multi-sequence
  drafting to enhance draft quality. Extensive experiments across seven tasks
  show that DuoDecoding achieves up to 2.61x speedup in generation latency, while
  reducing TTFT to 83% of that in conventional speculative decoding. The Code is
  available at https://github.com/KaiLv69/DuoDecoding.
paper_ai_summary: DuoDecoding optimizes speculative decoding by strategically deploying draft and target models to parallelize processes, reducing generation latency and time to first token.
paper_ai_keywords: LLMs autoregressive generation speculative decoding draft model target model CPU GPU parallel decoding hardware-aware optimal draft budget dynamic multi-sequence drafting TTFT