title: AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and
    Pruning
summary: Large language models (LLMs) have enabled the creation of multi-modal LLMs
  that exhibit strong comprehension of visual data such as images and videos.
  However, these models usually rely on extensive visual tokens from visual
  encoders, leading to high computational demands, which limits their
  applicability in resource-constrained environments and for long-context tasks.
  In this work, we propose a training-free adaptive inference method for
  multi-modal LLMs that can accommodate a broad range of efficiency requirements
  with a minimum performance drop. Our method consists of a) iterative token
  merging based on embedding similarity before LLMs, and b) progressive token
  pruning within LLM layers based on multi-modal importance. With a minimalist
  design, our method can be applied to both video and image LLMs. Extensive
  experiments on diverse video and image benchmarks demonstrate that, our method
  substantially reduces computation load (e.g., a 7-fold reduction in
  FLOPs) while preserving the performance of video and image LLMs. Further, under
  a similar computational cost, our method outperforms the state-of-the-art
  methods in long video understanding (e.g., +4.6 on MLVU).
  Additionally, our in-depth analysis provides insights into token redundancy and
  LLM layer behaviors, offering guidance for future research in designing
  efficient multi-modal LLMs. Our code will be available at
  https://github.com/LaVi-Lab/AIM.
paper_authors._id: 67510af7a9d103be6ce725e0
paper_authors.name: Yiwu Zhong
paper_authors._id: 67510af7a9d103be6ce725e1
paper_authors.name: Zhuoming Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:48:28.448Z
paper_authors._id: 666f2534a119281ee0e39783
paper_authors.avatarUrl: /avatars/1de737da5cdf68d9bb1f98db8d5ad8b9.svg
paper_authors.fullname: Zhuoming Liu
paper_authors.type: user
paper_authors.user: zhuomingliu
paper_authors._id: 67510af7a9d103be6ce725e2
paper_authors.name: Yin Li
paper_authors._id: 67510af7a9d103be6ce725e3
paper_authors.name: Liwei Wang
paper_title: AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and
    Pruning
paper_summary: Large language models (LLMs) have enabled the creation of multi-modal LLMs
  that exhibit strong comprehension of visual data such as images and videos.
  However, these models usually rely on extensive visual tokens from visual
  encoders, leading to high computational demands, which limits their
  applicability in resource-constrained environments and for long-context tasks.
  In this work, we propose a training-free adaptive inference method for
  multi-modal LLMs that can accommodate a broad range of efficiency requirements
  with a minimum performance drop. Our method consists of a) iterative token
  merging based on embedding similarity before LLMs, and b) progressive token
  pruning within LLM layers based on multi-modal importance. With a minimalist
  design, our method can be applied to both video and image LLMs. Extensive
  experiments on diverse video and image benchmarks demonstrate that, our method
  substantially reduces computation load (e.g., a 7-fold reduction in
  FLOPs) while preserving the performance of video and image LLMs. Further, under
  a similar computational cost, our method outperforms the state-of-the-art
  methods in long video understanding (e.g., +4.6 on MLVU).
  Additionally, our in-depth analysis provides insights into token redundancy and
  LLM layer behaviors, offering guidance for future research in designing
  efficient multi-modal LLMs. Our code will be available at
  https://github.com/LaVi-Lab/AIM.
paper_ai_summary: An adaptive inference method reduces computational demands in multi-modal LLMs for video and image understanding without significant performance loss.
paper_ai_keywords: large language models (LLMs) multi-modal LLMs iterative token merging embedding similarity progressive token pruning multi-modal importance FLOPs long video understanding token redundancy LLM layer behaviors