title: FastKV: KV Cache Compression for Fast Long-Context Processing with
    Token-Selective Propagation
summary: While large language models (LLMs) excel at handling long-context sequences,
  they require substantial key-value (KV) caches to store contextual information,
  which can heavily burden computational efficiency and memory usage. Previous
  efforts to compress these KV caches primarily focused on reducing memory
  demands but were limited in enhancing latency. To address this issue, we
  introduce FastKV, a KV cache compression method designed to enhance latency for
  long-context sequences. To enhance processing speeds while maintaining
  accuracy, FastKV adopts a novel Token-Selective Propagation (TSP) approach that
  retains the full context information in the initial layers of LLMs and
  selectively propagates only a portion of this information in deeper layers even
  in the prefill stage. Additionally, FastKV incorporates grouped-query attention
  (GQA)-aware KV cache compression to exploit the advantages of GQA in both
  memory and computational efficiency. Our experimental results show that FastKV
  achieves 2.00times and 1.40times improvements in time-to-first-token
  (TTFT) and throughput, respectively, compared to HeadKV, the state-of-the-art
  KV cache compression method. Moreover, FastKV successfully maintains accuracy
  on long-context benchmarks at levels comparable to the baselines. Our code is
  available at https://github.com/dongwonjo/FastKV.
paper_authors._id: 67a1a75f6aa8429da4945eeb
paper_authors.name: Dongwon Jo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:16.125Z
paper_authors._id: 639ffbc6beb95d698de9640d
paper_authors.avatarUrl: /avatars/7ef1aaadd5b378d00e17dc548e42cb7e.svg
paper_authors.fullname: Dongwon Jo
paper_authors.type: user
paper_authors.user: dongwonjo
paper_authors._id: 67a1a75f6aa8429da4945eec
paper_authors.name: Jiwon Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-04T09:39:14.253Z
paper_authors._id: 662672eaebdfec5cfdf1d034
paper_authors.avatarUrl: /avatars/61bc7add693c555e29ad3c1112215684.svg
paper_authors.fullname: Jiwon Song
paper_authors.type: user
paper_authors.user: jiwonsong
paper_authors._id: 67a1a75f6aa8429da4945eed
paper_authors.name: Yulhwa Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-04T17:35:45.706Z
paper_authors._id: 6566ddb96af53c602f80b1e2
paper_authors.avatarUrl: /avatars/403c8e486115920e50867b6462ddfd99.svg
paper_authors.fullname: Yulhwa Kim
paper_authors.type: user
paper_authors.user: YulhwaKim
paper_authors._id: 67a1a75f6aa8429da4945eee
paper_authors.name: Jae-Joon Kim
paper_title: FastKV: KV Cache Compression for Fast Long-Context Processing with
    Token-Selective Propagation
paper_summary: While large language models (LLMs) excel at handling long-context sequences,
  they require substantial key-value (KV) caches to store contextual information,
  which can heavily burden computational efficiency and memory usage. Previous
  efforts to compress these KV caches primarily focused on reducing memory
  demands but were limited in enhancing latency. To address this issue, we
  introduce FastKV, a KV cache compression method designed to enhance latency for
  long-context sequences. To enhance processing speeds while maintaining
  accuracy, FastKV adopts a novel Token-Selective Propagation (TSP) approach that
  retains the full context information in the initial layers of LLMs and
  selectively propagates only a portion of this information in deeper layers even
  in the prefill stage. Additionally, FastKV incorporates grouped-query attention
  (GQA)-aware KV cache compression to exploit the advantages of GQA in both
  memory and computational efficiency. Our experimental results show that FastKV
  achieves 2.00times and 1.40times improvements in time-to-first-token
  (TTFT) and throughput, respectively, compared to HeadKV, the state-of-the-art
  KV cache compression method. Moreover, FastKV successfully maintains accuracy
  on long-context benchmarks at levels comparable to the baselines. Our code is
  available at https://github.com/dongwonjo/FastKV.
paper_ai_summary: FastKV is a KV cache compression method for large language models that improves latency and throughput while maintaining accuracy through Token-Selective Propagation and grouped-query attention.
paper_ai_keywords: large language models KV cache computational efficiency memory usage Token-Selective Propagation grouped-query attention TSP GQA-aware KV cache compression time-to-first-token throughput long-context benchmarks