title: Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for
    Training-Free Acceleration
summary: To accelerate the inference of heavy Multimodal Large Language Models
  (MLLMs), this study rethinks the current landscape of training-free token
  reduction research. We regret to find that the critical components of existing
  methods are tightly intertwined, with their interconnections and effects
  remaining unclear for comparison, transfer, and expansion. Therefore, we
  propose a unified ''filter-correlate-compress'' paradigm that decomposes the
  token reduction into three distinct stages within a pipeline, maintaining
  consistent design objectives and elements while allowing for unique
  implementations. We additionally demystify the popular works and subsume them
  into our paradigm to showcase its universality. Finally, we offer a suite of
  methods grounded in the paradigm, striking a balance between speed and accuracy
  throughout different phases of the inference. Experimental results across 10
  benchmarks indicate that our methods can achieve up to an 82.4% reduction in
  FLOPs with a minimal impact on performance, simultaneously surpassing
  state-of-the-art training-free methods. Our project page is at
  https://ficoco-accelerate.github.io/.
paper_authors._id: 67469c4e599a83a637c9fa74
paper_authors.name: Yuhang Han
paper_authors._id: 67469c4e599a83a637c9fa75
paper_authors.name: Xuyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T14:47:56.691Z
paper_authors._id: 66a0caa1a7a6ed88ad1c0ddf
paper_authors.avatarUrl: /avatars/5bc46fd7cf7cfdb39c21fc1c6ce68597.svg
paper_authors.fullname: Xuyang Liu
paper_authors.type: user
paper_authors.user: xuyang-liu16
paper_authors._id: 67469c4e599a83a637c9fa76
paper_authors.name: Pengxiang Ding
paper_authors._id: 67469c4e599a83a637c9fa77
paper_authors.name: Donglin Wang
paper_authors._id: 67469c4e599a83a637c9fa78
paper_authors.name: Honggang Chen
paper_authors._id: 67469c4e599a83a637c9fa79
paper_authors.name: Qingsen Yan
paper_authors._id: 67469c4e599a83a637c9fa7a
paper_authors.name: Siteng Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:21:25.497Z
paper_authors._id: 65fd82762bf2cd20ddaa193f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yBYbWp_mT7UusYdkqtAvw.png
paper_authors.fullname: Siteng Huang
paper_authors.type: user
paper_authors.user: huangsiteng
paper_title: Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for
    Training-Free Acceleration
paper_summary: To accelerate the inference of heavy Multimodal Large Language Models
  (MLLMs), this study rethinks the current landscape of training-free token
  reduction research. We regret to find that the critical components of existing
  methods are tightly intertwined, with their interconnections and effects
  remaining unclear for comparison, transfer, and expansion. Therefore, we
  propose a unified ''filter-correlate-compress'' paradigm that decomposes the
  token reduction into three distinct stages within a pipeline, maintaining
  consistent design objectives and elements while allowing for unique
  implementations. We additionally demystify the popular works and subsume them
  into our paradigm to showcase its universality. Finally, we offer a suite of
  methods grounded in the paradigm, striking a balance between speed and accuracy
  throughout different phases of the inference. Experimental results across 10
  benchmarks indicate that our methods can achieve up to an 82.4% reduction in
  FLOPs with a minimal impact on performance, simultaneously surpassing
  state-of-the-art training-free methods. Our project page is at
  https://ficoco-accelerate.github.io/.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) token reduction filter-correlate-compress FLOPs