TidalDecode: Fast and Accurate LLM Decoding with Position Persistent
  Sparse Attention
Large language models (LLMs) have driven significant advancements across
diverse NLP tasks, with long-context models gaining prominence for handling
extended inputs. However, the expanding key-value (KV) cache size required by
Transformer architectures intensifies the memory constraints, particularly
during the decoding phase, creating a significant bottleneck. Existing sparse
attention mechanisms designed to address this bottleneck have two limitations:
(1) they often fail to reliably identify the most relevant tokens for
attention, and (2) they overlook the spatial coherence of token selection
across consecutive Transformer layers, which can lead to performance
degradation and substantial overhead in token selection. This paper introduces
TidalDecode, a simple yet effective algorithm and system for fast and accurate
LLM decoding through position persistent sparse attention. TidalDecode
leverages the spatial coherence of tokens selected by existing sparse attention
methods and introduces a few token selection layers that perform full attention
to identify the tokens with the highest attention scores, while all other
layers perform sparse attention with the pre-selected tokens. This design
enables TidalDecode to substantially reduce the overhead of token selection for
sparse attention without sacrificing the quality of the generated results.
Evaluation on a diverse set of LLMs and tasks shows that TidalDecode closely
matches the generative performance of full attention methods while reducing the
LLM decoding latency by up to 2.1x.
6706ad084c73cea935952f99
Lijie Yang
claimed_verified
2025-07-23T08:38:50.278Z
65d8523167819fdde61c9e9d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7CrWbLndFzd3YZDke1OgK.jpeg
Lijie Yang
user
drkylj
6706ad084c73cea935952f9a
Zhihao Zhang
extracted_pending
2024-10-09T16:19:21.988Z
6480ec2d856901b0edbbd47a
/avatars/dd8c22589c46fc19d7bf4a1329692d91.svg
Zhihao Jia
user
zhihaojia
6706ad084c73cea935952f9b
Zhuofu Chen
6706ad084c73cea935952f9c
Zikun Li
6706ad084c73cea935952f9d
Zhihao Jia
TidalDecode: Fast and Accurate LLM Decoding with Position Persistent
  Sparse Attention
Large language models (LLMs) have driven significant advancements across
diverse NLP tasks, with long-context models gaining prominence for handling
extended inputs. However, the expanding key-value (KV) cache size required by
Transformer architectures intensifies the memory constraints, particularly
during the decoding phase, creating a significant bottleneck. Existing sparse
attention mechanisms designed to address this bottleneck have two limitations:
(1) they often fail to reliably identify the most relevant tokens for
attention, and (2) they overlook the spatial coherence of token selection
across consecutive Transformer layers, which can lead to performance
degradation and substantial overhead in token selection. This paper introduces
TidalDecode, a simple yet effective algorithm and system for fast and accurate
LLM decoding through position persistent sparse attention. TidalDecode
leverages the spatial coherence of tokens selected by existing sparse attention
methods and introduces a few token selection layers that perform full attention
to identify the tokens with the highest attention scores, while all other
layers perform sparse attention with the pre-selected tokens. This design
enables TidalDecode to substantially reduce the overhead of token selection for
sparse attention without sacrificing the quality of the generated results.
Evaluation on a diverse set of LLMs and tasks shows that TidalDecode closely
matches the generative performance of full attention methods while reducing the
LLM decoding latency by up to 2.1x.
TidalDecode reduces LLM decoding latency by leveraging position persistent sparse attention and pre-selecting key tokens, improving performance without sacrificing quality.
LLMs long-context models Transformer architectures key-value cache sparse attention position persistent sparse attention