title: Lossless Acceleration of Large Language Models with Hierarchical
    Drafting based on Temporal Locality in Speculative Decoding
summary: Accelerating inference in Large Language Models (LLMs) is critical for
  real-time interactions, as they have been widely incorporated into real-world
  services. Speculative decoding, a fully algorithmic solution, has gained
  attention for improving inference speed by drafting and verifying tokens,
  thereby generating multiple tokens in a single forward pass. However, current
  drafting strategies usually require significant fine-tuning or have
  inconsistent performance across tasks. To address these challenges, we propose
  Hierarchy Drafting (HD), a novel lossless drafting approach that organizes
  various token sources into multiple databases in a hierarchical framework based
  on temporal locality. In the drafting step, HD sequentially accesses multiple
  databases to obtain draft tokens from the highest to the lowest locality,
  ensuring consistent acceleration across diverse tasks and minimizing drafting
  latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters
  demonstrate that HD outperforms existing database drafting methods, achieving
  robust inference speedups across model sizes, tasks, and temperatures.
paper_authors._id: 67aacaaaa03eecbc2d72835f
paper_authors.name: Sukmin Cho
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:54:43.377Z
paper_authors._id: 64ec4c04c782d648d28d70fc
paper_authors.avatarUrl: /avatars/6975526fcf4b513cc934b5bc45370a48.svg
paper_authors.fullname: Sukmin Cho
paper_authors.type: user
paper_authors.user: zomss
paper_authors._id: 67aacaaaa03eecbc2d728360
paper_authors.name: Sangjin Choi
paper_authors._id: 67aacaaaa03eecbc2d728361
paper_authors.name: Taeho Hwang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:54:45.737Z
paper_authors._id: 64d1e70a84f205869017703b
paper_authors.avatarUrl: /avatars/215d0d4db5f79cb74df4d888b18c6a0d.svg
paper_authors.fullname: Taeho Hwang
paper_authors.type: user
paper_authors.user: doubleyyh
paper_authors._id: 67aacaaaa03eecbc2d728362
paper_authors.name: Jeongyeon Seo
paper_authors._id: 67aacaaaa03eecbc2d728363
paper_authors.name: Soyeong Jeong
paper_authors._id: 67aacaaaa03eecbc2d728364
paper_authors.name: Huije Lee
paper_authors._id: 67aacaaaa03eecbc2d728365
paper_authors.name: Hoyun Song
paper_authors._id: 67aacaaaa03eecbc2d728366
paper_authors.name: Jong C. Park
paper_authors._id: 67aacaaaa03eecbc2d728367
paper_authors.name: Youngjin Kwon
paper_title: Lossless Acceleration of Large Language Models with Hierarchical
    Drafting based on Temporal Locality in Speculative Decoding
paper_summary: Accelerating inference in Large Language Models (LLMs) is critical for
  real-time interactions, as they have been widely incorporated into real-world
  services. Speculative decoding, a fully algorithmic solution, has gained
  attention for improving inference speed by drafting and verifying tokens,
  thereby generating multiple tokens in a single forward pass. However, current
  drafting strategies usually require significant fine-tuning or have
  inconsistent performance across tasks. To address these challenges, we propose
  Hierarchy Drafting (HD), a novel lossless drafting approach that organizes
  various token sources into multiple databases in a hierarchical framework based
  on temporal locality. In the drafting step, HD sequentially accesses multiple
  databases to obtain draft tokens from the highest to the lowest locality,
  ensuring consistent acceleration across diverse tasks and minimizing drafting
  latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters
  demonstrate that HD outperforms existing database drafting methods, achieving
  robust inference speedups across model sizes, tasks, and temperatures.
paper_ai_summary: Hierarchy Drafting improves inference speed in Large Language Models by organizing token sources into a hierarchical framework, ensuring consistent performance across various tasks.
paper_ai_keywords: Large Language Models LLMs speculative decoding forwarding pass Hierarchy Drafting HD hierarchical framework temporal locality draft tokens Spec-Bench