title: SnapKV: LLM Knows What You are Looking for Before Generation
summary: Large Language Models (LLMs) have made remarkable progress in processing
  extensive contexts, with the Key-Value (KV) cache playing a vital role in
  enhancing their performance. However, the growth of the KV cache in response to
  increasing input length poses challenges to memory and time efficiency. To
  address this problem, this paper introduces SnapKV, an innovative and
  fine-tuning-free approach that efficiently minimizes KV cache size while still
  delivering comparable performance in real-world applications.
    We discover that each attention head in the model consistently focuses on
  specific prompt attention features during generation. Meanwhile, this robust
  pattern can be obtained from an `observation' window located at the end of the
  prompts. Drawing on this insight, SnapKV automatically compresses KV caches by
  selecting clustered important KV positions for each attention head. Our
  approach significantly reduces the growing computational overhead and memory
  footprint when processing long input sequences. Specifically, SnapKV achieves a
  consistent decoding speed with a 3.6x increase in generation speed and an 8.2x
  enhancement in memory efficiency compared to baseline when processing inputs of
  16K tokens. At the same time, it maintains comparable performance to baseline
  models across 16 long sequence datasets. Moreover, SnapKV can process up to
  380K context tokens on a single A100-80GB GPU using HuggingFace implementation
  with minor changes, exhibiting only a negligible accuracy drop in the
  Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's
  potential for practical applications.
paper_authors._id: 662861ec650b96e3ab6ae49d
paper_authors.name: Yuhong Li
paper_authors._id: 662861ec650b96e3ab6ae49e
paper_authors.name: Yingbing Huang
paper_authors._id: 662861ec650b96e3ab6ae49f
paper_authors.name: Bowen Yang
paper_authors._id: 662861ec650b96e3ab6ae4a0
paper_authors.name: Bharat Venkitesh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:39:34.088Z
paper_authors._id: 65e8e96868cd34406483a639
paper_authors.avatarUrl: /avatars/77fbe9ebd1fccd2e15abb403e2139652.svg
paper_authors.fullname: Bharat venkitesh
paper_authors.type: user
paper_authors.user: bharatv007
paper_authors._id: 662861ec650b96e3ab6ae4a1
paper_authors.name: Acyr Locatelli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:39:39.937Z
paper_authors._id: 60d370b613f774189902f556
paper_authors.avatarUrl: /avatars/717bd7955f30eb76310d2074fd51afce.svg
paper_authors.fullname: Acyr Locatelli
paper_authors.type: user
paper_authors.user: acyrl
paper_authors._id: 662861ec650b96e3ab6ae4a2
paper_authors.name: Hanchen Ye
paper_authors._id: 662861ec650b96e3ab6ae4a3
paper_authors.name: Tianle Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-24T11:40:03.968Z
paper_authors._id: 64459357c525660aa20be337
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
paper_authors.fullname: Tianle Cai
paper_authors.type: user
paper_authors.user: tianlecai
paper_authors._id: 662861ec650b96e3ab6ae4a4
paper_authors.name: Patrick Lewis
paper_authors._id: 662861ec650b96e3ab6ae4a5
paper_authors.name: Deming Chen
paper_title: SnapKV: LLM Knows What You are Looking for Before Generation
paper_summary: Large Language Models (LLMs) have made remarkable progress in processing
  extensive contexts, with the Key-Value (KV) cache playing a vital role in
  enhancing their performance. However, the growth of the KV cache in response to
  increasing input length poses challenges to memory and time efficiency. To
  address this problem, this paper introduces SnapKV, an innovative and
  fine-tuning-free approach that efficiently minimizes KV cache size while still
  delivering comparable performance in real-world applications.
    We discover that each attention head in the model consistently focuses on
  specific prompt attention features during generation. Meanwhile, this robust
  pattern can be obtained from an `observation' window located at the end of the
  prompts. Drawing on this insight, SnapKV automatically compresses KV caches by
  selecting clustered important KV positions for each attention head. Our
  approach significantly reduces the growing computational overhead and memory
  footprint when processing long input sequences. Specifically, SnapKV achieves a
  consistent decoding speed with a 3.6x increase in generation speed and an 8.2x
  enhancement in memory efficiency compared to baseline when processing inputs of
  16K tokens. At the same time, it maintains comparable performance to baseline
  models across 16 long sequence datasets. Moreover, SnapKV can process up to
  380K context tokens on a single A100-80GB GPU using HuggingFace implementation
  with minor changes, exhibiting only a negligible accuracy drop in the
  Needle-in-a-Haystack test. Further comprehensive studies suggest SnapKV's
  potential for practical applications.
paper_ai_keywords: Key-Value (KV) cache attention head prompt attention features observation window attention head KV positions decoding speed generation speed memory efficiency