Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Models
<|reference_start|>Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Models: Vision-Language Large Models (VLMs) recently become primary backbone of AI, due to the impressive performance. However, their expensive computation costs, i.e., throughput and delay, impede potentials in the real-world scenarios. To achieve acceleration for VLMs, most existing methods focus on the model perspective: pruning, distillation, quantization, but completely overlook the data-perspective redundancy. To fill the overlook, this paper pioneers the severity of data redundancy, and designs one plug-and-play Turbo module guided by information degree to prune inefficient tokens from visual or textual data. In pursuit of efficiency-performance trade-offs, information degree takes two crucial factors into consideration: mutual redundancy and semantic value. Concretely, the former evaluates data duplication between sequential tokens; while the latter evaluates each token by its contribution to the overall semantics. As a result, tokens with high information degree carry less redundancy and stronger semantics. For VLMs' calculation, Turbo works as a user-friendly plug-in that sorts data referring to information degree, utilizing only top-level ones to save costs. Its advantages are multifaceted, e.g., being generally compatible to various VLMs across understanding and generation, simple use without re-training and trivial engineering efforts. On multiple VLMs benchmarks, we fully experiment to demonstrate the good acceleration of Turbo, under negligible performance drop.<|reference_end|>
@article{ju2024turbo:,
    title={Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language
  Large Models},
    author={Chen Ju, Haicheng Wang, Haozhe Cheng, Xu Chen, Zhonghua Zhai, Weilin
  Huang, Jinsong Lan, Shuai Xiao, Bo Zheng},
    journal={arXiv preprint arXiv:2407.11717},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.11717},
    primaryClass={cs.CV}
}