CoreMatching: A Co-adaptive Sparse Inference Framework with Token and
  Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
Vision-Language Models (VLMs) excel across diverse tasks but suffer from high
inference costs in time and memory. Token sparsity mitigates inefficiencies in
token usage, while neuron sparsity reduces high-dimensional computations, both
offering promising solutions to enhance efficiency. Recently, these two
sparsity paradigms have evolved largely in parallel, fostering the prevailing
assumption that they function independently. However, a fundamental yet
underexplored question remains: Do they truly operate in isolation, or is there
a deeper underlying interplay that has yet to be uncovered? In this paper, we
conduct the first comprehensive investigation into this question. By
introducing and analyzing the matching mechanism between Core Neurons and Core
Tokens, we found that key neurons and tokens for inference mutually influence
and reinforce each other. Building on this insight, we propose CoreMatching, a
co-adaptive sparse inference framework, which leverages the synergy between
token and neuron sparsity to enhance inference efficiency. Through theoretical
analysis and efficiency evaluations, we demonstrate that the proposed method
surpasses state-of-the-art baselines on ten image understanding tasks and three
hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs
reduction and a 10x overall speedup. Code is released at
https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.
68373a5d8209dbdcf52d9723
Qinsi Wang
claimed_verified
2025-06-23T08:32:28.997Z
66128e8b7e0e7a64652dbbdf
/avatars/b72ea5b14b55ff3af920c06b69a60b3f.svg
Wang
user
Qinsi1
68373a5d8209dbdcf52d9724
Hancheng Ye
claimed_verified
2025-06-07T05:49:53.215Z
6597f70d260bcd5fb5529613
/avatars/8fd7a210f3cb0de1229b49a3cd1ad8a0.svg
Hancheng Ye
user
HankYe
68373a5d8209dbdcf52d9725
Ming-Yu Chung
68373a5d8209dbdcf52d9726
Yudong Liu
68373a5d8209dbdcf52d9727
Yueqian Lin
claimed_verified
2025-06-06T07:42:20.930Z
64b5198c25882acb62fb77ef
https://cdn-avatars.huggingface.co/v1/production/uploads/64b5198c25882acb62fb77ef/HX9pfMEPQlfjvSAgSLplY.png
Yueqian Lin
user
linyueqian
68373a5d8209dbdcf52d9728
Martin Kuo
68373a5d8209dbdcf52d9729
Mingyuan Ma
68373a5d8209dbdcf52d972a
Jianyi Zhang
68373a5d8209dbdcf52d972b
Yiran Chen
CoreMatching: A Co-adaptive Sparse Inference Framework with Token and
  Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
Vision-Language Models (VLMs) excel across diverse tasks but suffer from high
inference costs in time and memory. Token sparsity mitigates inefficiencies in
token usage, while neuron sparsity reduces high-dimensional computations, both
offering promising solutions to enhance efficiency. Recently, these two
sparsity paradigms have evolved largely in parallel, fostering the prevailing
assumption that they function independently. However, a fundamental yet
underexplored question remains: Do they truly operate in isolation, or is there
a deeper underlying interplay that has yet to be uncovered? In this paper, we
conduct the first comprehensive investigation into this question. By
introducing and analyzing the matching mechanism between Core Neurons and Core
Tokens, we found that key neurons and tokens for inference mutually influence
and reinforce each other. Building on this insight, we propose CoreMatching, a
co-adaptive sparse inference framework, which leverages the synergy between
token and neuron sparsity to enhance inference efficiency. Through theoretical
analysis and efficiency evaluations, we demonstrate that the proposed method
surpasses state-of-the-art baselines on ten image understanding tasks and three
hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs
reduction and a 10x overall speedup. Code is released at
https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main.
A core-matching framework enhances inference efficiency in vision-language models by leveraging the synergy between token and neuron sparsity, outperforming baselines across multiple tasks and devices.
vision-language models token sparsity neuron sparsity inference efficiency core neurons core tokens 协同adaptive sparse inference framework FLOPs reduction