title: CoreMatching: A Co-adaptive Sparse Inference Framework with Token and
    Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
summary: Vision-Language Models (VLMs) excel across diverse tasks but suffer from high
  inference costs in time and memory. Token sparsity mitigates inefficiencies in
  token usage, while neuron sparsity reduces high-dimensional computations, both
  offering promising solutions to enhance efficiency. Recently, these two
  sparsity paradigms have evolved largely in parallel, fostering the prevailing
  assumption that they function independently. However, a fundamental yet
  underexplored question remains: Do they truly operate in isolation, or is there
  a deeper underlying interplay that has yet to be uncovered? In this paper, we
  conduct the first comprehensive investigation into this question. By
  introducing and analyzing the matching mechanism between Core Neurons and Core
  Tokens, we found that key neurons and tokens for inference mutually influence
  and reinforce each other. Building on this insight, we propose CoreMatching, a
  co-adaptive sparse inference framework, which leverages the synergy between
  token and neuron sparsity to enhance inference efficiency. Through theoretical
  analysis and efficiency evaluations, we demonstrate that the proposed method
  surpasses state-of-the-art baselines on ten image understanding tasks and three
  hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs
  reduction and a 10x overall speedup. Code is released at
  https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.
paper_authors._id: 68373a5d8209dbdcf52d9723
paper_authors.name: Qinsi Wang
paper_authors._id: 68373a5d8209dbdcf52d9724
paper_authors.name: Hancheng Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:49:53.215Z
paper_authors._id: 6597f70d260bcd5fb5529613
paper_authors.avatarUrl: /avatars/8fd7a210f3cb0de1229b49a3cd1ad8a0.svg
paper_authors.fullname: Hancheng Ye
paper_authors.type: user
paper_authors.user: HankYe
paper_authors._id: 68373a5d8209dbdcf52d9725
paper_authors.name: Ming-Yu Chung
paper_authors._id: 68373a5d8209dbdcf52d9726
paper_authors.name: Yudong Liu
paper_authors._id: 68373a5d8209dbdcf52d9727
paper_authors.name: Yueqian Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:42:20.930Z
paper_authors._id: 64b5198c25882acb62fb77ef
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b5198c25882acb62fb77ef/HX9pfMEPQlfjvSAgSLplY.png
paper_authors.fullname: Yueqian Lin
paper_authors.type: user
paper_authors.user: linyueqian
paper_authors._id: 68373a5d8209dbdcf52d9728
paper_authors.name: Martin Kuo
paper_authors._id: 68373a5d8209dbdcf52d9729
paper_authors.name: Mingyuan Ma
paper_authors._id: 68373a5d8209dbdcf52d972a
paper_authors.name: Jianyi Zhang
paper_authors._id: 68373a5d8209dbdcf52d972b
paper_authors.name: Yiran Chen
paper_title: CoreMatching: A Co-adaptive Sparse Inference Framework with Token and
    Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
paper_summary: Vision-Language Models (VLMs) excel across diverse tasks but suffer from high
  inference costs in time and memory. Token sparsity mitigates inefficiencies in
  token usage, while neuron sparsity reduces high-dimensional computations, both
  offering promising solutions to enhance efficiency. Recently, these two
  sparsity paradigms have evolved largely in parallel, fostering the prevailing
  assumption that they function independently. However, a fundamental yet
  underexplored question remains: Do they truly operate in isolation, or is there
  a deeper underlying interplay that has yet to be uncovered? In this paper, we
  conduct the first comprehensive investigation into this question. By
  introducing and analyzing the matching mechanism between Core Neurons and Core
  Tokens, we found that key neurons and tokens for inference mutually influence
  and reinforce each other. Building on this insight, we propose CoreMatching, a
  co-adaptive sparse inference framework, which leverages the synergy between
  token and neuron sparsity to enhance inference efficiency. Through theoretical
  analysis and efficiency evaluations, we demonstrate that the proposed method
  surpasses state-of-the-art baselines on ten image understanding tasks and three
  hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs
  reduction and a 10x overall speedup. Code is released at
  https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.
paper_ai_summary: A core-matching framework enhances inference efficiency in vision-language models by leveraging the synergy between token and neuron sparsity, outperforming baselines across multiple tasks and devices.
paper_ai_keywords: vision-language models token sparsity neuron sparsity inference efficiency core neurons core tokens 协同adaptive sparse inference framework FLOPs reduction