title: LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior
    Accuracy Preservation
summary: We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV
  Cache in large language model (LLM) inference, delivering substantial memory
  savings while preserving superior performance. Previous methods either assume
  that later tokens are more important or attempt to predict important tokens
  based on earlier attention patterns. Both approaches, however, can result in
  performance bottlenecks or frequent mispredictions.
    LogQuant takes a different approach. By applying a log-based filtering
  mechanism, it selectively compresses the KV Cache across the entire context,
  achieving better performance with the same or even reduced memory footprint
  compared to existing methods. In benchmark tests, it enhances throughput by 25%
  and boosts batch size by 60% without increasing memory consumption. For
  challenging tasks such as Math and Code Completion, LogQuant improves accuracy
  by 40% to 200% at the same compression ratio, outperforming comparable
  techniques.LogQuant integrates effortlessly with popular inference frameworks
  like Python's transformers library. Implementation can be available in
  https://github.com/Concyclics/LogQuantKV.
paper_authors._id: 67e4b27cfe1f5acc68028de9
paper_authors.name: Han Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-27T09:03:29.747Z
paper_authors._id: 6399c67bf78f75ae73146760
paper_authors.avatarUrl: /avatars/d1c3b0eff67b598a1ad58e297382957f.svg
paper_authors.fullname: CHEN Han
paper_authors.type: user
paper_authors.user: Concyclics
paper_authors._id: 67e4b27cfe1f5acc68028dea
paper_authors.name: Zicong Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:02:00.596Z
paper_authors._id: 650ccf6a36ac7eba06ea1cfa
paper_authors.avatarUrl: /avatars/50374fca4f6cc2cf7a6601cd8d3f725b.svg
paper_authors.fullname: Zicong Jiang
paper_authors.type: user
paper_authors.user: Zicong99
paper_authors._id: 67e4b27cfe1f5acc68028deb
paper_authors.name: Zining Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-27T10:02:07.675Z
paper_authors._id: 64b781c5da8017900e7b8b25
paper_authors.avatarUrl: /avatars/0db9a83b0908cc6b9417360ed77fcc1a.svg
paper_authors.fullname: zining zhang
paper_authors.type: user
paper_authors.user: deciding
paper_authors._id: 67e4b27cfe1f5acc68028dec
paper_authors.name: Bingsheng He
paper_authors._id: 67e4b27cfe1f5acc68028ded
paper_authors.name: Pingyi Luo
paper_authors._id: 67e4b27cfe1f5acc68028dee
paper_authors.name: Mian Lu
paper_authors._id: 67e4b27cfe1f5acc68028def
paper_authors.name: Yuqiang Chen
paper_title: LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior
    Accuracy Preservation
paper_summary: We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV
  Cache in large language model (LLM) inference, delivering substantial memory
  savings while preserving superior performance. Previous methods either assume
  that later tokens are more important or attempt to predict important tokens
  based on earlier attention patterns. Both approaches, however, can result in
  performance bottlenecks or frequent mispredictions.
    LogQuant takes a different approach. By applying a log-based filtering
  mechanism, it selectively compresses the KV Cache across the entire context,
  achieving better performance with the same or even reduced memory footprint
  compared to existing methods. In benchmark tests, it enhances throughput by 25%
  and boosts batch size by 60% without increasing memory consumption. For
  challenging tasks such as Math and Code Completion, LogQuant improves accuracy
  by 40% to 200% at the same compression ratio, outperforming comparable
  techniques.LogQuant integrates effortlessly with popular inference frameworks
  like Python's transformers library. Implementation can be available in
  https://github.com/Concyclics/LogQuantKV.
paper_ai_summary: LogQuant, a 2-bit quantization technique for KV Cache in LLM inference, enhances performance and memory efficiency compared to existing methods.
paper_ai_keywords: 2-bit quantization KV Cache large language model (LLM) log-based filtering mechanism