title: Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language
    Models
summary: Video large language models (VideoLLMs) have demonstrated the capability to
  process longer video inputs and enable complex reasoning and analysis. However,
  due to the thousands of visual tokens from the video frames, key-value (KV)
  cache can significantly increase memory requirements, becoming a bottleneck for
  inference speed and memory usage. KV cache quantization is a widely used
  approach to address this problem. In this paper, we find that 2-bit KV
  quantization of VideoLLMs can hardly hurt the model performance, while the
  limit of KV cache quantization in even lower bits has not been investigated. To
  bridge this gap, we introduce VidKV, a plug-and-play KV cache quantization
  method to compress the KV cache to lower than 2 bits. Specifically, (1) for
  key, we propose a mixed-precision quantization strategy in the channel
  dimension, where we perform 2-bit quantization for anomalous channels and 1-bit
  quantization combined with FFT for normal channels; (2) for value, we implement
  1.58-bit quantization while selectively filtering semantically salient visual
  tokens for targeted preservation, for a better trade-off between precision and
  model performance. Importantly, our findings suggest that the value cache of
  VideoLLMs should be quantized in a per-channel fashion instead of the per-token
  fashion proposed by prior KV cache quantization works for LLMs. Empirically,
  extensive results with LLaVA-OV-7B and Qwen2.5-VL-7B on six benchmarks show
  that VidKV effectively compresses the KV cache to 1.5-bit and 1.58-bit
  precision with almost no performance drop compared to the FP16 counterparts.
paper_authors._id: 67dd2cbabf4c007db3bc0b76
paper_authors.name: Keda Tao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:37:38.025Z
paper_authors._id: 65ce28c6340c3e914285aa58
paper_authors.avatarUrl: /avatars/ffaa6d6ce92274bff960f8ea229a37f8.svg
paper_authors.fullname: Keda TAO
paper_authors.type: user
paper_authors.user: KD-TAO
paper_authors._id: 67dd2cbabf4c007db3bc0b77
paper_authors.name: Haoxuan You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:59:36.878Z
paper_authors._id: 65284ba3e3b3844f5e487a0b
paper_authors.avatarUrl: /avatars/fce12a233bddb611d441f25a2ab17725.svg
paper_authors.fullname: Haoxuan You
paper_authors.type: user
paper_authors.user: HaoxuanYou
paper_authors._id: 67dd2cbabf4c007db3bc0b78
paper_authors.name: Yang Sui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:04:13.545Z
paper_authors._id: 63787b13500186f250ba377c
paper_authors.avatarUrl: /avatars/96bb6051662109e9cd25e6df7d738f17.svg
paper_authors.fullname: yangsui
paper_authors.type: user
paper_authors.user: yangsui
paper_authors._id: 67dd2cbabf4c007db3bc0b79
paper_authors.name: Can Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:59:48.645Z
paper_authors._id: 66a9bdfc35c89692442ba4b7
paper_authors.avatarUrl: /avatars/ad87d1d2d81775eeb0920bf3ebe08cc2.svg
paper_authors.fullname: Can Qin
paper_authors.type: user
paper_authors.user: canqin001
paper_authors._id: 67dd2cbabf4c007db3bc0b7a
paper_authors.name: Huan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T10:11:29.992Z
paper_authors._id: 62b624f3b52bef716e248fd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
paper_authors.fullname: Huan Wang
paper_authors.type: user
paper_authors.user: Huan-WhoRegisteredMyName
paper_title: Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language
    Models
paper_summary: Video large language models (VideoLLMs) have demonstrated the capability to
  process longer video inputs and enable complex reasoning and analysis. However,
  due to the thousands of visual tokens from the video frames, key-value (KV)
  cache can significantly increase memory requirements, becoming a bottleneck for
  inference speed and memory usage. KV cache quantization is a widely used
  approach to address this problem. In this paper, we find that 2-bit KV
  quantization of VideoLLMs can hardly hurt the model performance, while the
  limit of KV cache quantization in even lower bits has not been investigated. To
  bridge this gap, we introduce VidKV, a plug-and-play KV cache quantization
  method to compress the KV cache to lower than 2 bits. Specifically, (1) for
  key, we propose a mixed-precision quantization strategy in the channel
  dimension, where we perform 2-bit quantization for anomalous channels and 1-bit
  quantization combined with FFT for normal channels; (2) for value, we implement
  1.58-bit quantization while selectively filtering semantically salient visual
  tokens for targeted preservation, for a better trade-off between precision and
  model performance. Importantly, our findings suggest that the value cache of
  VideoLLMs should be quantized in a per-channel fashion instead of the per-token
  fashion proposed by prior KV cache quantization works for LLMs. Empirically,
  extensive results with LLaVA-OV-7B and Qwen2.5-VL-7B on six benchmarks show
  that VidKV effectively compresses the KV cache to 1.5-bit and 1.58-bit
  precision with almost no performance drop compared to the FP16 counterparts.
paper_ai_summary: VidKV, a new KV cache quantization method for VideoLLMs, compresses the KV cache to below 2 bits with minimal impact on performance by employing mixed-precision and channel-wise quantization techniques.
paper_ai_keywords: KV cache video frames visual tokens KV cache quantization VidKV mixed-precision quantization FFT per-channel quantization semantically salient visual tokens LLaVA-OV-7B Qwen2.5-VL-7B FP16