Finch: Prompt-guided Key-Value Cache Compression
Recent large language model applications, such as Retrieval-Augmented
Generation and chatbots, have led to an increased need to process longer input
contexts. However, this requirement is hampered by inherent limitations.
Architecturally, models are constrained by a context window defined during
training. Additionally, processing extensive texts requires substantial GPU
memory. We propose a novel approach, Finch, to compress the input context by
leveraging the pre-trained model weights of the self-attention. Given a prompt
and a long text, Finch iteratively identifies the most relevant Key (K) and
Value (V) pairs over chunks of the text conditioned on the prompt. Only such
pairs are stored in the KV cache, which, within the space constrained by the
context window, ultimately contains a compressed version of the long text. Our
proposal enables models to consume large inputs even with high compression (up
to 93x) while preserving semantic integrity without the need for fine-tuning.
66aca4c3f4b511a1e9724578
Giulio Corallo
claimed_verified
2024-08-02T09:21:48.085Z
62c88e75a5ac2974c0a5c8ea
https://cdn-avatars.huggingface.co/v1/production/uploads/1666106981514-62c88e75a5ac2974c0a5c8ea.jpeg
Giulio Corallo
user
giulio98
66aca4c3f4b511a1e9724579
Paolo Papotti
extracted_confirmed
2025-03-11T07:35:23.296Z
67cfe779293bf2fc0c7163f2
/avatars/6d719eafd43a214d34d61bc6d9908511.svg
Paolo Papotti
user
papotti
Finch: Prompt-guided Key-Value Cache Compression
Recent large language model applications, such as Retrieval-Augmented
Generation and chatbots, have led to an increased need to process longer input
contexts. However, this requirement is hampered by inherent limitations.
Architecturally, models are constrained by a context window defined during
training. Additionally, processing extensive texts requires substantial GPU
memory. We propose a novel approach, Finch, to compress the input context by
leveraging the pre-trained model weights of the self-attention. Given a prompt
and a long text, Finch iteratively identifies the most relevant Key (K) and
Value (V) pairs over chunks of the text conditioned on the prompt. Only such
pairs are stored in the KV cache, which, within the space constrained by the
context window, ultimately contains a compressed version of the long text. Our
proposal enables models to consume large inputs even with high compression (up
to 93x) while preserving semantic integrity without the need for fine-tuning.
Finch enables large input processing in models by compressing context via pre-trained self-attention weights, maintaining semantic integrity without fine-tuning.
self-attention Key (K) Value (V) KV cache context window compression