FastKV: KV Cache Compression for Fast Long-Context Processing with
  Token-Selective Propagation
While large language models (LLMs) excel at handling long-context sequences,
they require substantial key-value (KV) caches to store contextual information,
which can heavily burden computational efficiency and memory usage. Previous
efforts to compress these KV caches primarily focused on reducing memory
demands but were limited in enhancing latency. To address this issue, we
introduce FastKV, a KV cache compression method designed to enhance latency for
long-context sequences. To enhance processing speeds while maintaining
accuracy, FastKV adopts a novel Token-Selective Propagation (TSP) approach that
retains the full context information in the initial layers of LLMs and
selectively propagates only a portion of this information in deeper layers even
in the prefill stage. Additionally, FastKV incorporates grouped-query attention
(GQA)-aware KV cache compression to exploit the advantages of GQA in both
memory and computational efficiency. Our experimental results show that FastKV
achieves 2.00times and 1.40times improvements in time-to-first-token
(TTFT) and throughput, respectively, compared to HeadKV, the state-of-the-art
KV cache compression method. Moreover, FastKV successfully maintains accuracy
on long-context benchmarks at levels comparable to the baselines. Our code is
available at https://github.com/dongwonjo/FastKV.
67a1a75f6aa8429da4945eeb
Dongwon Jo
claimed_verified
2025-02-04T09:39:16.125Z
639ffbc6beb95d698de9640d
/avatars/7ef1aaadd5b378d00e17dc548e42cb7e.svg
Dongwon Jo
user
dongwonjo
67a1a75f6aa8429da4945eec
Jiwon Song
claimed_verified
2025-02-04T09:39:14.253Z
662672eaebdfec5cfdf1d034
/avatars/61bc7add693c555e29ad3c1112215684.svg
Jiwon Song
user
jiwonsong
67a1a75f6aa8429da4945eed
Yulhwa Kim
admin_assigned
2025-02-04T17:35:45.706Z
6566ddb96af53c602f80b1e2
/avatars/403c8e486115920e50867b6462ddfd99.svg
Yulhwa Kim
user
YulhwaKim
67a1a75f6aa8429da4945eee
Jae-Joon Kim
FastKV: KV Cache Compression for Fast Long-Context Processing with
  Token-Selective Propagation
While large language models (LLMs) excel at handling long-context sequences,
they require substantial key-value (KV) caches to store contextual information,
which can heavily burden computational efficiency and memory usage. Previous
efforts to compress these KV caches primarily focused on reducing memory
demands but were limited in enhancing latency. To address this issue, we
introduce FastKV, a KV cache compression method designed to enhance latency for
long-context sequences. To enhance processing speeds while maintaining
accuracy, FastKV adopts a novel Token-Selective Propagation (TSP) approach that
retains the full context information in the initial layers of LLMs and
selectively propagates only a portion of this information in deeper layers even
in the prefill stage. Additionally, FastKV incorporates grouped-query attention
(GQA)-aware KV cache compression to exploit the advantages of GQA in both
memory and computational efficiency. Our experimental results show that FastKV
achieves 2.00times and 1.40times improvements in time-to-first-token
(TTFT) and throughput, respectively, compared to HeadKV, the state-of-the-art
KV cache compression method. Moreover, FastKV successfully maintains accuracy
on long-context benchmarks at levels comparable to the baselines. Our code is
available at https://github.com/dongwonjo/FastKV.
FastKV is a KV cache compression method for large language models that improves latency and throughput while maintaining accuracy through Token-Selective Propagation and grouped-query attention.
large language models KV cache computational efficiency memory usage Token-Selective Propagation grouped-query attention TSP GQA-aware KV cache compression time-to-first-token throughput long-context benchmarks