SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference
<|reference_start|>SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference: In vision-language models (VLMs), visual tokens usually consume a significant amount of computational overhead, despite their sparser information density compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens and require additional training data. Differently, we propose an efficient training-free token optimization mechanism dubbed SparseVLM without extra parameters or fine-tuning costs. Concretely, given that visual tokens complement text tokens in VLMs for linguistic reasoning, we select visual-relevant text tokens to rate the significance of vision tokens within the self-attention matrix extracted from the VLMs. Then we progressively prune irrelevant tokens. To maximize sparsity while retaining essential information, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that our SparseVLM improves the efficiency of various VLMs across a range of image and video understanding tasks. In particular, LLaVA equipped with SparseVLM reduces 61% to 67% FLOPs with a compression ratio of 78% while maintaining 93% of the accuracy. Our code is available at https://github.com/Gumpest/SparseVLMs.<|reference_end|>
@article{zhang2024sparsevlm:,
    title={SparseVLM: Visual Token Sparsification for Efficient Vision-Language
  Model Inference},
    author={Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan
  Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, Shanghang
  Zhang},
    journal={arXiv preprint arXiv:2410.04417},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.04417},
    primaryClass={cs.CV}
}