Lossless Acceleration of Large Language Models with Hierarchical
  Drafting based on Temporal Locality in Speculative Decoding
Accelerating inference in Large Language Models (LLMs) is critical for
real-time interactions, as they have been widely incorporated into real-world
services. Speculative decoding, a fully algorithmic solution, has gained
attention for improving inference speed by drafting and verifying tokens,
thereby generating multiple tokens in a single forward pass. However, current
drafting strategies usually require significant fine-tuning or have
inconsistent performance across tasks. To address these challenges, we propose
Hierarchy Drafting (HD), a novel lossless drafting approach that organizes
various token sources into multiple databases in a hierarchical framework based
on temporal locality. In the drafting step, HD sequentially accesses multiple
databases to obtain draft tokens from the highest to the lowest locality,
ensuring consistent acceleration across diverse tasks and minimizing drafting
latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters
demonstrate that HD outperforms existing database drafting methods, achieving
robust inference speedups across model sizes, tasks, and temperatures.
67aacaaaa03eecbc2d72835f
Sukmin Cho
claimed_verified
2025-02-11T07:54:43.377Z
64ec4c04c782d648d28d70fc
/avatars/6975526fcf4b513cc934b5bc45370a48.svg
Sukmin Cho
user
zomss
67aacaaaa03eecbc2d728360
Sangjin Choi
67aacaaaa03eecbc2d728361
Taeho Hwang
claimed_verified
2025-02-11T07:54:45.737Z
64d1e70a84f205869017703b
/avatars/215d0d4db5f79cb74df4d888b18c6a0d.svg
Taeho Hwang
user
doubleyyh
67aacaaaa03eecbc2d728362
Jeongyeon Seo
67aacaaaa03eecbc2d728363
Soyeong Jeong
67aacaaaa03eecbc2d728364
Huije Lee
67aacaaaa03eecbc2d728365
Hoyun Song
67aacaaaa03eecbc2d728366
Jong C. Park
67aacaaaa03eecbc2d728367
Youngjin Kwon
Lossless Acceleration of Large Language Models with Hierarchical
  Drafting based on Temporal Locality in Speculative Decoding
Accelerating inference in Large Language Models (LLMs) is critical for
real-time interactions, as they have been widely incorporated into real-world
services. Speculative decoding, a fully algorithmic solution, has gained
attention for improving inference speed by drafting and verifying tokens,
thereby generating multiple tokens in a single forward pass. However, current
drafting strategies usually require significant fine-tuning or have
inconsistent performance across tasks. To address these challenges, we propose
Hierarchy Drafting (HD), a novel lossless drafting approach that organizes
various token sources into multiple databases in a hierarchical framework based
on temporal locality. In the drafting step, HD sequentially accesses multiple
databases to obtain draft tokens from the highest to the lowest locality,
ensuring consistent acceleration across diverse tasks and minimizing drafting
latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters
demonstrate that HD outperforms existing database drafting methods, achieving
robust inference speedups across model sizes, tasks, and temperatures.
Hierarchy Drafting improves inference speed in Large Language Models by organizing token sources into a hierarchical framework, ensuring consistent performance across various tasks.
Large Language Models LLMs speculative decoding forwarding pass Hierarchy Drafting HD hierarchical framework temporal locality draft tokens Spec-Bench