LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior
  Accuracy Preservation
We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV
Cache in large language model (LLM) inference, delivering substantial memory
savings while preserving superior performance. Previous methods either assume
that later tokens are more important or attempt to predict important tokens
based on earlier attention patterns. Both approaches, however, can result in
performance bottlenecks or frequent mispredictions.
  LogQuant takes a different approach. By applying a log-based filtering
mechanism, it selectively compresses the KV Cache across the entire context,
achieving better performance with the same or even reduced memory footprint
compared to existing methods. In benchmark tests, it enhances throughput by 25%
and boosts batch size by 60% without increasing memory consumption. For
challenging tasks such as Math and Code Completion, LogQuant improves accuracy
by 40% to 200% at the same compression ratio, outperforming comparable
techniques.LogQuant integrates effortlessly with popular inference frameworks
like Python's transformers library. Implementation can be available in
https://github.com/Concyclics/LogQuantKV.
67e4b27cfe1f5acc68028de9
Han Chen
claimed_verified
2025-03-27T09:03:29.747Z
6399c67bf78f75ae73146760
/avatars/d1c3b0eff67b598a1ad58e297382957f.svg
CHEN Han
user
Concyclics
67e4b27cfe1f5acc68028dea
Zicong Jiang
admin_assigned
2025-03-27T10:02:00.596Z
650ccf6a36ac7eba06ea1cfa
/avatars/50374fca4f6cc2cf7a6601cd8d3f725b.svg
Zicong Jiang
user
Zicong99
67e4b27cfe1f5acc68028deb
Zining Zhang
admin_assigned
2025-03-27T10:02:07.675Z
64b781c5da8017900e7b8b25
/avatars/0db9a83b0908cc6b9417360ed77fcc1a.svg
zining zhang
user
deciding
67e4b27cfe1f5acc68028dec
Bingsheng He
67e4b27cfe1f5acc68028ded
Pingyi Luo
67e4b27cfe1f5acc68028dee
Mian Lu
67e4b27cfe1f5acc68028def
Yuqiang Chen
LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior
  Accuracy Preservation
We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV
Cache in large language model (LLM) inference, delivering substantial memory
savings while preserving superior performance. Previous methods either assume
that later tokens are more important or attempt to predict important tokens
based on earlier attention patterns. Both approaches, however, can result in
performance bottlenecks or frequent mispredictions.
  LogQuant takes a different approach. By applying a log-based filtering
mechanism, it selectively compresses the KV Cache across the entire context,
achieving better performance with the same or even reduced memory footprint
compared to existing methods. In benchmark tests, it enhances throughput by 25%
and boosts batch size by 60% without increasing memory consumption. For
challenging tasks such as Math and Code Completion, LogQuant improves accuracy
by 40% to 200% at the same compression ratio, outperforming comparable
techniques.LogQuant integrates effortlessly with popular inference frameworks
like Python's transformers library. Implementation can be available in
https://github.com/Concyclics/LogQuantKV.
LogQuant, a 2-bit quantization technique for KV Cache in LLM inference, enhances performance and memory efficiency compared to existing methods.
2-bit quantization KV Cache large language model (LLM) log-based filtering mechanism