title: ZipVL: Efficient Large Vision-Language Models with Dynamic Token
    Sparsification and KV Cache Compression
summary: The efficiency of large vision-language models (LVLMs) is constrained by the
  computational bottleneck of the attention mechanism during the prefill phase
  and the memory bottleneck of fetching the key-value (KV) cache in the decoding
  phase, particularly in scenarios involving high-resolution images or videos.
  Visual content often exhibits substantial redundancy, resulting in highly
  sparse attention maps within LVLMs. This sparsity can be leveraged to
  accelerate attention computation or compress the KV cache through various
  approaches. However, most studies focus on addressing only one of these
  bottlenecks and do not adequately support dynamic adjustment of sparsity
  concerning distinct layers or tasks. In this paper, we present ZipVL, an
  efficient inference framework designed for LVLMs that resolves both computation
  and memory bottlenecks through a dynamic ratio allocation strategy of important
  tokens. This ratio is adaptively determined based on the layer-specific
  distribution of attention scores, rather than fixed hyper-parameters, thereby
  improving efficiency for less complex tasks while maintaining high performance
  for more challenging ones. Then we select important tokens based on their
  normalized attention scores and perform attention mechanism solely on those
  important tokens to accelerate the prefill phase. To mitigate the memory
  bottleneck in the decoding phase, we employ mixed-precision quantization to the
  KV cache, where high-bit quantization is used for caches of important tokens,
  while low-bit quantization is applied to those of less importance. Our
  experiments demonstrate that ZipVL can accelerate the prefill phase by
  2.6times and reduce GPU memory usage by 50.0%, with a minimal accuracy
  reduction of only 0.2% on Video-MME benchmark over LongVA-7B model, effectively
  enhancing the generation efficiency of LVLMs.
paper_authors._id: 670ca3f0977ff8095c05cb73
paper_authors.name: Yefei He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T11:16:52.423Z
paper_authors._id: 65a88c3d26598b995531fff1
paper_authors.avatarUrl: /avatars/b1a524857d8572d0405476661b434160.svg
paper_authors.fullname: Yefei He
paper_authors.type: user
paper_authors.user: yefly
paper_authors._id: 670ca3f0977ff8095c05cb74
paper_authors.name: Feng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T11:16:54.546Z
paper_authors._id: 64574380f3ef144c0e69d484
paper_authors.avatarUrl: /avatars/a0a84757cb0bf09c24291803e1389b49.svg
paper_authors.fullname: Feng Chen
paper_authors.type: user
paper_authors.user: chenfeng1271
paper_authors._id: 670ca3f0977ff8095c05cb75
paper_authors.name: Jing Liu
paper_authors._id: 670ca3f0977ff8095c05cb76
paper_authors.name: Wenqi Shao
paper_authors._id: 670ca3f0977ff8095c05cb77
paper_authors.name: Hong Zhou
paper_authors._id: 670ca3f0977ff8095c05cb78
paper_authors.name: Kaipeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:27:33.696Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_authors._id: 670ca3f0977ff8095c05cb79
paper_authors.name: Bohan Zhuang
paper_title: ZipVL: Efficient Large Vision-Language Models with Dynamic Token
    Sparsification and KV Cache Compression
paper_summary: The efficiency of large vision-language models (LVLMs) is constrained by the
  computational bottleneck of the attention mechanism during the prefill phase
  and the memory bottleneck of fetching the key-value (KV) cache in the decoding
  phase, particularly in scenarios involving high-resolution images or videos.
  Visual content often exhibits substantial redundancy, resulting in highly
  sparse attention maps within LVLMs. This sparsity can be leveraged to
  accelerate attention computation or compress the KV cache through various
  approaches. However, most studies focus on addressing only one of these
  bottlenecks and do not adequately support dynamic adjustment of sparsity
  concerning distinct layers or tasks. In this paper, we present ZipVL, an
  efficient inference framework designed for LVLMs that resolves both computation
  and memory bottlenecks through a dynamic ratio allocation strategy of important
  tokens. This ratio is adaptively determined based on the layer-specific
  distribution of attention scores, rather than fixed hyper-parameters, thereby
  improving efficiency for less complex tasks while maintaining high performance
  for more challenging ones. Then we select important tokens based on their
  normalized attention scores and perform attention mechanism solely on those
  important tokens to accelerate the prefill phase. To mitigate the memory
  bottleneck in the decoding phase, we employ mixed-precision quantization to the
  KV cache, where high-bit quantization is used for caches of important tokens,
  while low-bit quantization is applied to those of less importance. Our
  experiments demonstrate that ZipVL can accelerate the prefill phase by
  2.6times and reduce GPU memory usage by 50.0%, with a minimal accuracy
  reduction of only 0.2% on Video-MME benchmark over LongVA-7B model, effectively
  enhancing the generation efficiency of LVLMs.
paper_ai_keywords: LVLMs (Large Vision-Language Models) attention mechanism prefill phase memory bottleneck key-value (KV) cache decoding phase redundancy attention maps sparsity ZipVL dynamic ratio allocation important tokens attention scores mixed-precision quantization high-bit quantization low-bit quantization Video-MME benchmark LongVA-7B model