title: SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
summary: Multimodal Large Language Models (MLLMs) are commonly derived by extending
  pre-trained Large Language Models (LLMs) with visual capabilities. In this
  work, we investigate how MLLMs process visual inputs by analyzing their
  attention mechanisms. We reveal a surprising sparsity phenomenon: only a small
  subset (approximately less than 5%) of attention heads in LLMs actively
  contribute to visual understanding, termed visual heads. To identify these
  heads efficiently, we design a training-free framework that quantifies
  head-level visual relevance through targeted response analysis. Building on
  this discovery, we introduce SparseMM, a KV-Cache optimization strategy that
  allocates asymmetric computation budgets to heads in LLMs based on their visual
  scores, leveraging the sparity of visual heads for accelerating the inference
  of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the
  particularity of visual, SparseMM prioritizes stress and retaining visual
  semantics during decoding. Extensive evaluations across mainstream multimodal
  benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency
  trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52%
  memory reduction during generation while maintaining performance parity on
  efficiency test. Our project is open sourced at
  https://github.com/CR400AF-A/SparseMM.
paper_authors._id: 68424fe9bdc448822b31beac
paper_authors.name: Jiahui Wang
paper_authors._id: 68424fe9bdc448822b31bead
paper_authors.name: Zuyan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:41:17.520Z
paper_authors._id: 64f001bfabd9fb1914398bd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
paper_authors.fullname: liuzuyan
paper_authors.type: user
paper_authors.user: Zuyan
paper_authors._id: 68424fe9bdc448822b31beae
paper_authors.name: Yongming Rao
paper_authors._id: 68424fe9bdc448822b31beaf
paper_authors.name: Jiwen Lu
paper_title: SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs
paper_summary: Multimodal Large Language Models (MLLMs) are commonly derived by extending
  pre-trained Large Language Models (LLMs) with visual capabilities. In this
  work, we investigate how MLLMs process visual inputs by analyzing their
  attention mechanisms. We reveal a surprising sparsity phenomenon: only a small
  subset (approximately less than 5%) of attention heads in LLMs actively
  contribute to visual understanding, termed visual heads. To identify these
  heads efficiently, we design a training-free framework that quantifies
  head-level visual relevance through targeted response analysis. Building on
  this discovery, we introduce SparseMM, a KV-Cache optimization strategy that
  allocates asymmetric computation budgets to heads in LLMs based on their visual
  scores, leveraging the sparity of visual heads for accelerating the inference
  of MLLMs. Compared with prior KV-Cache acceleration methods that ignore the
  particularity of visual, SparseMM prioritizes stress and retaining visual
  semantics during decoding. Extensive evaluations across mainstream multimodal
  benchmarks demonstrate that SparseMM achieves superior accuracy-efficiency
  trade-offs. Notably, SparseMM delivers 1.38x real-time acceleration and 52%
  memory reduction during generation while maintaining performance parity on
  efficiency test. Our project is open sourced at
  https://github.com/CR400AF-A/SparseMM.
paper_ai_summary: MLLLMs achieve enhanced efficiency through SparseMM, a KV-Cache optimization strategy that identifies and prioritizes visual heads, leading to significant real-time acceleration and memory reduction without compromising performance.
paper_ai_keywords: multimodal large language models LLMs visual capabilities attention mechanisms visual heads targeted response analysis KV-Cache optimization SparseMM head-level visual relevance visual semantics multimodal benchmarks