SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
Multimodal Large Language Models (MLLMs) are commonly derived by extending
pre-trained Large Language Models (LLMs) with visual capabilities. In this
work, we investigate how MLLMs process visual inputs by analyzing their
attention mechanisms. We reveal a surprising sparsity phenomenon: only a small
subset (approximately less than 5%) of attention heads in LLMs actively
contribute to visual understanding, termed visual heads. To identify these
heads efficiently, we design a training-free framework that quantifies
head-level visual relevance through targeted response analysis. Building on
this discovery, we introduce SparseMM, a KV-Cache optimization strategy that
allocates asymmetric computation budgets to heads in LLMs based on their visual
scores, leveraging the sparity of visual heads for accelerating the inference
of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the
particularity of visual, SparseMM prioritizes stress and retaining visual
semantics during decoding. Extensive evaluations across mainstream multimodal
benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency
trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52%
memory reduction during generation while maintaining performance parity on
efficiency test. Our project is open sourced at
https://github.com/CR400AF-A/SparseMM.
68424fe9bdc448822b31beac
Jiahui Wang
68424fe9bdc448822b31bead
Zuyan Liu
claimed_verified
2025-06-06T07:41:17.520Z
64f001bfabd9fb1914398bd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
liuzuyan
user
Zuyan
68424fe9bdc448822b31beae
Yongming Rao
68424fe9bdc448822b31beaf
Jiwen Lu
SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
Multimodal Large Language Models (MLLMs) are commonly derived by extending
pre-trained Large Language Models (LLMs) with visual capabilities. In this
work, we investigate how MLLMs process visual inputs by analyzing their
attention mechanisms. We reveal a surprising sparsity phenomenon: only a small
subset (approximately less than 5%) of attention heads in LLMs actively
contribute to visual understanding, termed visual heads. To identify these
heads efficiently, we design a training-free framework that quantifies
head-level visual relevance through targeted response analysis. Building on
this discovery, we introduce SparseMM, a KV-Cache optimization strategy that
allocates asymmetric computation budgets to heads in LLMs based on their visual
scores, leveraging the sparity of visual heads for accelerating the inference
of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the
particularity of visual, SparseMM prioritizes stress and retaining visual
semantics during decoding. Extensive evaluations across mainstream multimodal
benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency
trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52%
memory reduction during generation while maintaining performance parity on
efficiency test. Our project is open sourced at
https://github.com/CR400AF-A/SparseMM.
MLLLMs achieve enhanced efficiency through SparseMM, a KV-Cache optimization strategy that identifies and prioritizes visual heads, leading to significant real-time acceleration and memory reduction without compromising performance.
multimodal large language models LLMs visual capabilities attention mechanisms visual heads targeted response analysis KV-Cache optimization SparseMM head-level visual relevance visual semantics multimodal benchmarks