title: Finch: Prompt-guided Key-Value Cache Compression
summary: Recent large language model applications, such as Retrieval-Augmented
  Generation and chatbots, have led to an increased need to process longer input
  contexts. However, this requirement is hampered by inherent limitations.
  Architecturally, models are constrained by a context window defined during
  training. Additionally, processing extensive texts requires substantial GPU
  memory. We propose a novel approach, Finch, to compress the input context by
  leveraging the pre-trained model weights of the self-attention. Given a prompt
  and a long text, Finch iteratively identifies the most relevant Key (K) and
  Value (V) pairs over chunks of the text conditioned on the prompt. Only such
  pairs are stored in the KV cache, which, within the space constrained by the
  context window, ultimately contains a compressed version of the long text. Our
  proposal enables models to consume large inputs even with high compression (up
  to 93x) while preserving semantic integrity without the need for fine-tuning.
paper_authors._id: 66aca4c3f4b511a1e9724578
paper_authors.name: Giulio Corallo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-02T09:21:48.085Z
paper_authors._id: 62c88e75a5ac2974c0a5c8ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666106981514-62c88e75a5ac2974c0a5c8ea.jpeg
paper_authors.fullname: Giulio Corallo
paper_authors.type: user
paper_authors.user: giulio98
paper_authors._id: 66aca4c3f4b511a1e9724579
paper_authors.name: Paolo Papotti
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-11T07:35:23.296Z
paper_authors._id: 67cfe779293bf2fc0c7163f2
paper_authors.avatarUrl: /avatars/6d719eafd43a214d34d61bc6d9908511.svg
paper_authors.fullname: Paolo Papotti
paper_authors.type: user
paper_authors.user: papotti
paper_title: Finch: Prompt-guided Key-Value Cache Compression
paper_summary: Recent large language model applications, such as Retrieval-Augmented
  Generation and chatbots, have led to an increased need to process longer input
  contexts. However, this requirement is hampered by inherent limitations.
  Architecturally, models are constrained by a context window defined during
  training. Additionally, processing extensive texts requires substantial GPU
  memory. We propose a novel approach, Finch, to compress the input context by
  leveraging the pre-trained model weights of the self-attention. Given a prompt
  and a long text, Finch iteratively identifies the most relevant Key (K) and
  Value (V) pairs over chunks of the text conditioned on the prompt. Only such
  pairs are stored in the KV cache, which, within the space constrained by the
  context window, ultimately contains a compressed version of the long text. Our
  proposal enables models to consume large inputs even with high compression (up
  to 93x) while preserving semantic integrity without the need for fine-tuning.
paper_ai_summary: Finch enables large input processing in models by compressing context via pre-trained self-attention weights, maintaining semantic integrity without fine-tuning.
paper_ai_keywords: self-attention Key (K) Value (V) KV cache context window compression