title: Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge
    Reasoning
summary: Incorporating external knowledge in large language models (LLMs) enhances
  their utility across diverse applications, but existing methods have
  trade-offs. Retrieval-Augmented Generation (RAG) fetches evidence via
  similarity search, but key information may fall outside top ranked results.
  Long-context models can process multiple documents but are computationally
  expensive and limited by context window size. Inspired by students condensing
  study material for open-book exams, we propose task-aware key-value (KV) cache
  compression, which compresses external knowledge in a zero- or few-shot setup.
  This enables LLMs to reason efficiently over a compacted representation of all
  relevant information. Experiments show our approach outperforms both RAG and
  task-agnostic compression methods. On LongBench v2, it improves accuracy by up
  to 7 absolute points over RAG with a 30x compression rate, while reducing
  inference latency from 0.43s to 0.16s. A synthetic dataset highlights that RAG
  performs well when sparse evidence suffices, whereas task-aware compression is
  superior for broad knowledge tasks.
paper_authors._id: 67d01f99a47b429698a2b34e
paper_authors.name: Giulio Corallo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T11:43:02.035Z
paper_authors._id: 62c88e75a5ac2974c0a5c8ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666106981514-62c88e75a5ac2974c0a5c8ea.jpeg
paper_authors.fullname: Giulio Corallo
paper_authors.type: user
paper_authors.user: giulio98
paper_authors._id: 67d01f99a47b429698a2b34f
paper_authors.name: Orion Weller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:17:49.755Z
paper_authors._id: 6362d9712691058b19de1ba4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6362d9712691058b19de1ba4/c9QrA2oE6lcs_46ShaTY1.jpeg
paper_authors.fullname: Orion Weller
paper_authors.type: user
paper_authors.user: orionweller
paper_authors._id: 67d01f99a47b429698a2b350
paper_authors.name: Fabio Petroni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:17:56.241Z
paper_authors._id: 6707ed4d371a647d6c8b5f04
paper_authors.avatarUrl: /avatars/e9cb4c9caa3943588891e41e7f62cd69.svg
paper_authors.fullname: Fabio Petroni
paper_authors.type: user
paper_authors.user: fabiopetroni
paper_authors._id: 67d01f99a47b429698a2b351
paper_authors.name: Paolo Papotti
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-03-11T12:35:45.283Z
paper_authors._id: 67cfe779293bf2fc0c7163f2
paper_authors.avatarUrl: /avatars/6d719eafd43a214d34d61bc6d9908511.svg
paper_authors.fullname: Paolo Papotti
paper_authors.type: user
paper_authors.user: papotti
paper_title: Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge
    Reasoning
paper_summary: Incorporating external knowledge in large language models (LLMs) enhances
  their utility across diverse applications, but existing methods have
  trade-offs. Retrieval-Augmented Generation (RAG) fetches evidence via
  similarity search, but key information may fall outside top ranked results.
  Long-context models can process multiple documents but are computationally
  expensive and limited by context window size. Inspired by students condensing
  study material for open-book exams, we propose task-aware key-value (KV) cache
  compression, which compresses external knowledge in a zero- or few-shot setup.
  This enables LLMs to reason efficiently over a compacted representation of all
  relevant information. Experiments show our approach outperforms both RAG and
  task-agnostic compression methods. On LongBench v2, it improves accuracy by up
  to 7 absolute points over RAG with a 30x compression rate, while reducing
  inference latency from 0.43s to 0.16s. A synthetic dataset highlights that RAG
  performs well when sparse evidence suffices, whereas task-aware compression is
  superior for broad knowledge tasks.
paper_ai_keywords: Retrieval-Augmented Generation (RAG) key-value (KV) cache compression zero-shot setup few-shot setup LongBench v2 inference latency