TriForce: Lossless Acceleration of Long Sequence Generation with
  Hierarchical Speculative Decoding
With large language models (LLMs) widely deployed in long content generation
recently, there has emerged an increasing demand for efficient long-sequence
inference support. However, key-value (KV) cache, which is stored to avoid
re-computation, has emerged as a critical bottleneck by growing linearly in
size with the sequence length. Due to the auto-regressive nature of LLMs, the
entire KV cache will be loaded for every generated token, resulting in low
utilization of computational cores and high latency. While various compression
methods for KV cache have been proposed to alleviate this issue, they suffer
from degradation in generation quality. We introduce TriForce, a hierarchical
speculative decoding system that is scalable to long sequence generation. This
approach leverages the original model weights and dynamic sparse KV cache via
retrieval as a draft model, which serves as an intermediate layer in the
hierarchy and is further speculated by a smaller model to reduce its drafting
latency. TriForce not only facilitates impressive speedups for Llama2-7B-128K,
achieving up to 2.31times on an A100 GPU but also showcases scalability in
handling even longer contexts. For the offloading setting on two RTX 4090 GPUs,
TriForce achieves 0.108s/tokenx2014only half as slow as the
auto-regressive baseline on an A100, which attains 7.78times on our
optimized offloading system. Additionally, TriForce performs 4.86times than
DeepSpeed-Zero-Inference on a single RTX 4090 GPU. TriForce's robustness is
highlighted by its consistently outstanding performance across various
temperatures. The code is available at
https://github.com/Infini-AI-Lab/TriForce.
6621cae311c923d051faf1ff
Hanshi Sun
claimed_verified
2024-04-19T13:34:16.128Z
651b23a0a6e00a1678c9d8af
https://cdn-avatars.huggingface.co/v1/production/uploads/651b23a0a6e00a1678c9d8af/_Ip-ysvcjGpFCI9HtcTgx.jpeg
Hanshi
user
preminstrel
6621cae311c923d051faf200
Zhuoming Chen
claimed_verified
2025-06-06T07:44:14.144Z
650ddecfdf31b1809f12a273
/avatars/2c6300471cd09a067d56aa09bc06ad78.svg
ZMC
user
ZMC2019
6621cae311c923d051faf201
Xinyu Yang
extracted_pending
2024-04-19T01:37:40.260Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
6621cae311c923d051faf202
Yuandong Tian
admin_assigned
2024-04-19T08:24:43.627Z
6344cf73ee1504dbcd5bdfe7
/avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
Yuandong Tian
user
tydsh
6621cae311c923d051faf203
Beidi Chen
admin_assigned
2024-04-19T08:24:53.636Z
64b732f832403871593e082c
/avatars/dd21932b0c167131ee7545a622c46c3c.svg
Beidi Chen
user
beidic
TriForce: Lossless Acceleration of Long Sequence Generation with
  Hierarchical Speculative Decoding
With large language models (LLMs) widely deployed in long content generation
recently, there has emerged an increasing demand for efficient long-sequence
inference support. However, key-value (KV) cache, which is stored to avoid
re-computation, has emerged as a critical bottleneck by growing linearly in
size with the sequence length. Due to the auto-regressive nature of LLMs, the
entire KV cache will be loaded for every generated token, resulting in low
utilization of computational cores and high latency. While various compression
methods for KV cache have been proposed to alleviate this issue, they suffer
from degradation in generation quality. We introduce TriForce, a hierarchical
speculative decoding system that is scalable to long sequence generation. This
approach leverages the original model weights and dynamic sparse KV cache via
retrieval as a draft model, which serves as an intermediate layer in the
hierarchy and is further speculated by a smaller model to reduce its drafting
latency. TriForce not only facilitates impressive speedups for Llama2-7B-128K,
achieving up to 2.31times on an A100 GPU but also showcases scalability in
handling even longer contexts. For the offloading setting on two RTX 4090 GPUs,
TriForce achieves 0.108s/tokenx2014only half as slow as the
auto-regressive baseline on an A100, which attains 7.78times on our
optimized offloading system. Additionally, TriForce performs 4.86times than
DeepSpeed-Zero-Inference on a single RTX 4090 GPU. TriForce's robustness is
highlighted by its consistently outstanding performance across various
temperatures. The code is available at
https://github.com/Infini-AI-Lab/TriForce.
TriForce, a hierarchical speculative decoding system, accelerates long-sequence inference for large language models by using a dynamic sparse KV cache and layered model speculation, achieving significant speedups compared to baseline methods.
LLMs long content generation key-value cache auto-regressive compression methods speculative decoding dynamic sparse KV cache retrieval draft model smaller model Llama2-7B-128K RTX 4090 A100 GPU DeepSpeed-Zero-Inference