Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for
  Training-Free Acceleration
To accelerate the inference of heavy Multimodal Large Language Models
(MLLMs), this study rethinks the current landscape of training-free token
reduction research. We regret to find that the critical components of existing
methods are tightly intertwined, with their interconnections and effects
remaining unclear for comparison, transfer, and expansion. Therefore, we
propose a unified ''filter-correlate-compress'' paradigm that decomposes the
token reduction into three distinct stages within a pipeline, maintaining
consistent design objectives and elements while allowing for unique
implementations. We additionally demystify the popular works and subsume them
into our paradigm to showcase its universality. Finally, we offer a suite of
methods grounded in the paradigm, striking a balance between speed and accuracy
throughout different phases of the inference. Experimental results across 10
benchmarks indicate that our methods can achieve up to an 82.4% reduction in
FLOPs with a minimal impact on performance, simultaneously surpassing
state-of-the-art training-free methods. Our project page is at
https://ficoco-accelerate.github.io/.
67469c4e599a83a637c9fa74
Yuhang Han
67469c4e599a83a637c9fa75
Xuyang Liu
claimed_verified
2024-12-09T14:47:56.691Z
66a0caa1a7a6ed88ad1c0ddf
https://cdn-avatars.huggingface.co/v1/production/uploads/66a0caa1a7a6ed88ad1c0ddf/WoOP24-ruuHy4ryNhRp0D.jpeg
Xuyang Liu
user
xuyang-liu16
67469c4e599a83a637c9fa76
Pengxiang Ding
67469c4e599a83a637c9fa77
Donglin Wang
67469c4e599a83a637c9fa78
Honggang Chen
67469c4e599a83a637c9fa79
Qingsen Yan
67469c4e599a83a637c9fa7a
Siteng Huang
claimed_verified
2024-12-11T09:21:25.497Z
65fd82762bf2cd20ddaa193f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yBYbWp_mT7UusYdkqtAvw.png
Siteng Huang
user
huangsiteng
Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for
  Training-Free Acceleration
To accelerate the inference of heavy Multimodal Large Language Models
(MLLMs), this study rethinks the current landscape of training-free token
reduction research. We regret to find that the critical components of existing
methods are tightly intertwined, with their interconnections and effects
remaining unclear for comparison, transfer, and expansion. Therefore, we
propose a unified ''filter-correlate-compress'' paradigm that decomposes the
token reduction into three distinct stages within a pipeline, maintaining
consistent design objectives and elements while allowing for unique
implementations. We additionally demystify the popular works and subsume them
into our paradigm to showcase its universality. Finally, we offer a suite of
methods grounded in the paradigm, striking a balance between speed and accuracy
throughout different phases of the inference. Experimental results across 10
benchmarks indicate that our methods can achieve up to an 82.4% reduction in
FLOPs with a minimal impact on performance, simultaneously surpassing
state-of-the-art training-free methods. Our project page is at
https://ficoco-accelerate.github.io/.
A unified filter-correlate-compress paradigm accelerates inference in Multimodal Large Language Models with minimal impact on performance.
filter-correlate-compress token reduction inference acceleration FLOPs Multimodal Large Language Models