title: TidalDecode: Fast and Accurate LLM Decoding with Position Persistent
    Sparse Attention
summary: Large language models (LLMs) have driven significant advancements across
  diverse NLP tasks, with long-context models gaining prominence for handling
  extended inputs. However, the expanding key-value (KV) cache size required by
  Transformer architectures intensifies the memory constraints, particularly
  during the decoding phase, creating a significant bottleneck. Existing sparse
  attention mechanisms designed to address this bottleneck have two limitations:
  (1) they often fail to reliably identify the most relevant tokens for
  attention, and (2) they overlook the spatial coherence of token selection
  across consecutive Transformer layers, which can lead to performance
  degradation and substantial overhead in token selection. This paper introduces
  TidalDecode, a simple yet effective algorithm and system for fast and accurate
  LLM decoding through position persistent sparse attention. TidalDecode
  leverages the spatial coherence of tokens selected by existing sparse attention
  methods and introduces a few token selection layers that perform full attention
  to identify the tokens with the highest attention scores, while all other
  layers perform sparse attention with the pre-selected tokens. This design
  enables TidalDecode to substantially reduce the overhead of token selection for
  sparse attention without sacrificing the quality of the generated results.
  Evaluation on a diverse set of LLMs and tasks shows that TidalDecode closely
  matches the generative performance of full attention methods while reducing the
  LLM decoding latency by up to 2.1x.
paper_authors._id: 6706ad084c73cea935952f99
paper_authors.name: Lijie Yang
paper_authors._id: 6706ad084c73cea935952f9a
paper_authors.name: Zhihao Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-09T16:19:21.988Z
paper_authors._id: 6480ec2d856901b0edbbd47a
paper_authors.avatarUrl: /avatars/dd8c22589c46fc19d7bf4a1329692d91.svg
paper_authors.fullname: Zhihao Jia
paper_authors.type: user
paper_authors.user: zhihaojia
paper_authors._id: 6706ad084c73cea935952f9b
paper_authors.name: Zhuofu Chen
paper_authors._id: 6706ad084c73cea935952f9c
paper_authors.name: Zikun Li
paper_authors._id: 6706ad084c73cea935952f9d
paper_authors.name: Zhihao Jia
paper_title: TidalDecode: Fast and Accurate LLM Decoding with Position Persistent
    Sparse Attention
paper_summary: Large language models (LLMs) have driven significant advancements across
  diverse NLP tasks, with long-context models gaining prominence for handling
  extended inputs. However, the expanding key-value (KV) cache size required by
  Transformer architectures intensifies the memory constraints, particularly
  during the decoding phase, creating a significant bottleneck. Existing sparse
  attention mechanisms designed to address this bottleneck have two limitations:
  (1) they often fail to reliably identify the most relevant tokens for
  attention, and (2) they overlook the spatial coherence of token selection
  across consecutive Transformer layers, which can lead to performance
  degradation and substantial overhead in token selection. This paper introduces
  TidalDecode, a simple yet effective algorithm and system for fast and accurate
  LLM decoding through position persistent sparse attention. TidalDecode
  leverages the spatial coherence of tokens selected by existing sparse attention
  methods and introduces a few token selection layers that perform full attention
  to identify the tokens with the highest attention scores, while all other
  layers perform sparse attention with the pre-selected tokens. This design
  enables TidalDecode to substantially reduce the overhead of token selection for
  sparse attention without sacrificing the quality of the generated results.
  Evaluation on a diverse set of LLMs and tasks shows that TidalDecode closely
  matches the generative performance of full attention methods while reducing the
  LLM decoding latency by up to 2.1x.
paper_ai_summary: TidalDecode reduces LLM decoding latency by leveraging position persistent sparse attention and pre-selecting key tokens, improving performance without sacrificing quality.
paper_ai_keywords: LLMs long-context models Transformer architectures key-value cache sparse attention position persistent sparse attention