title: TriForce: Lossless Acceleration of Long Sequence Generation with
    Hierarchical Speculative Decoding
summary: With large language models (LLMs) widely deployed in long content generation
  recently, there has emerged an increasing demand for efficient long-sequence
  inference support. However, key-value (KV) cache, which is stored to avoid
  re-computation, has emerged as a critical bottleneck by growing linearly in
  size with the sequence length. Due to the auto-regressive nature of LLMs, the
  entire KV cache will be loaded for every generated token, resulting in low
  utilization of computational cores and high latency. While various compression
  methods for KV cache have been proposed to alleviate this issue, they suffer
  from degradation in generation quality. We introduce TriForce, a hierarchical
  speculative decoding system that is scalable to long sequence generation. This
  approach leverages the original model weights and dynamic sparse KV cache via
  retrieval as a draft model, which serves as an intermediate layer in the
  hierarchy and is further speculated by a smaller model to reduce its drafting
  latency. TriForce not only facilitates impressive speedups for Llama2-7B-128K,
  achieving up to 2.31times on an A100 GPU but also showcases scalability in
  handling even longer contexts. For the offloading setting on two RTX 4090 GPUs,
  TriForce achieves 0.108s/tokenx2014only half as slow as the
  auto-regressive baseline on an A100, which attains 7.78times on our
  optimized offloading system. Additionally, TriForce performs 4.86times than
  DeepSpeed-Zero-Inference on a single RTX 4090 GPU. TriForce's robustness is
  highlighted by its consistently outstanding performance across various
  temperatures. The code is available at
  https://github.com/Infini-AI-Lab/TriForce.
paper_authors._id: 6621cae311c923d051faf1ff
paper_authors.name: Hanshi Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T13:34:16.128Z
paper_authors._id: 651b23a0a6e00a1678c9d8af
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651b23a0a6e00a1678c9d8af/_Ip-ysvcjGpFCI9HtcTgx.jpeg
paper_authors.fullname: Hanshi
paper_authors.type: user
paper_authors.user: preminstrel
paper_authors._id: 6621cae311c923d051faf200
paper_authors.name: Zhuoming Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:44:14.144Z
paper_authors._id: 650ddecfdf31b1809f12a273
paper_authors.avatarUrl: /avatars/2c6300471cd09a067d56aa09bc06ad78.svg
paper_authors.fullname: ZMC
paper_authors.type: user
paper_authors.user: ZMC2019
paper_authors._id: 6621cae311c923d051faf201
paper_authors.name: Xinyu Yang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-19T01:37:40.260Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 6621cae311c923d051faf202
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:24:43.627Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 6621cae311c923d051faf203
paper_authors.name: Beidi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:24:53.636Z
paper_authors._id: 64b732f832403871593e082c
paper_authors.avatarUrl: /avatars/dd21932b0c167131ee7545a622c46c3c.svg
paper_authors.fullname: Beidi Chen
paper_authors.type: user
paper_authors.user: beidic
paper_title: TriForce: Lossless Acceleration of Long Sequence Generation with
    Hierarchical Speculative Decoding
paper_summary: With large language models (LLMs) widely deployed in long content generation
  recently, there has emerged an increasing demand for efficient long-sequence
  inference support. However, key-value (KV) cache, which is stored to avoid
  re-computation, has emerged as a critical bottleneck by growing linearly in
  size with the sequence length. Due to the auto-regressive nature of LLMs, the
  entire KV cache will be loaded for every generated token, resulting in low
  utilization of computational cores and high latency. While various compression
  methods for KV cache have been proposed to alleviate this issue, they suffer
  from degradation in generation quality. We introduce TriForce, a hierarchical
  speculative decoding system that is scalable to long sequence generation. This
  approach leverages the original model weights and dynamic sparse KV cache via
  retrieval as a draft model, which serves as an intermediate layer in the
  hierarchy and is further speculated by a smaller model to reduce its drafting
  latency. TriForce not only facilitates impressive speedups for Llama2-7B-128K,
  achieving up to 2.31times on an A100 GPU but also showcases scalability in
  handling even longer contexts. For the offloading setting on two RTX 4090 GPUs,
  TriForce achieves 0.108s/tokenx2014only half as slow as the
  auto-regressive baseline on an A100, which attains 7.78times on our
  optimized offloading system. Additionally, TriForce performs 4.86times than
  DeepSpeed-Zero-Inference on a single RTX 4090 GPU. TriForce's robustness is
  highlighted by its consistently outstanding performance across various
  temperatures. The code is available at
  https://github.com/Infini-AI-Lab/TriForce.
paper_ai_summary: TriForce, a hierarchical speculative decoding system, accelerates long-sequence inference for large language models by using a dynamic sparse KV cache and layered model speculation, achieving significant speedups compared to baseline methods.
paper_ai_keywords: LLMs long content generation key-value cache auto-regressive compression methods speculative decoding dynamic sparse KV cache retrieval draft model smaller model Llama2-7B-128K RTX 4090 A100 GPU DeepSpeed-Zero-Inference