title: HoliTom: Holistic Token Merging for Fast Video Large Language Models
summary: Video large language models (video LLMs) excel at video comprehension but
  face significant computational inefficiency due to redundant video tokens.
  Existing token pruning methods offer solutions. However, approaches operating
  within the LLM (inner-LLM pruning), such as FastV, incur intrinsic
  computational overhead in shallow layers. In contrast, methods performing token
  pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy
  within individual frames or limited temporal windows, neglecting the crucial
  global temporal dynamics and correlations across longer video sequences. This
  leads to sub-optimal spatio-temporal reduction and does not leverage video
  compressibility fully. Crucially, the synergistic potential and mutual
  influence of combining these strategies remain unexplored. To further reduce
  redundancy, we introduce HoliTom, a novel training-free holistic token merging
  framework. HoliTom employs outer-LLM pruning through global redundancy-aware
  temporal segmentation, followed by spatial-temporal merging to reduce visual
  tokens by over 90%, significantly alleviating the LLM's computational burden.
  Complementing this, we introduce a robust inner-LLM token similarity-based
  merging approach, designed for superior performance and compatibility with
  outer-LLM pruning. Evaluations demonstrate our method's promising
  efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational
  costs to 6.9% of FLOPs while maintaining 99.1% of the original performance.
  Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a
  1.32x acceleration in decoding throughput, highlighting the practical benefits
  of our integrated pruning approach for efficient video LLMs inference.
paper_authors._id: 6836a5a9bec1d6dbb3e10454
paper_authors.name: Kele Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:56:19.978Z
paper_authors._id: 6696755fd26a65bd255184d3
paper_authors.avatarUrl: /avatars/8d46c21a7b23f0100a7e3385fea61edf.svg
paper_authors.fullname: Kele Shao
paper_authors.type: user
paper_authors.user: keleshao
paper_authors._id: 6836a5a9bec1d6dbb3e10455
paper_authors.name: Keda Tao
paper_authors._id: 6836a5a9bec1d6dbb3e10456
paper_authors.name: Can Qin
paper_authors._id: 6836a5a9bec1d6dbb3e10457
paper_authors.name: Haoxuan You
paper_authors._id: 6836a5a9bec1d6dbb3e10458
paper_authors.name: Yang Sui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:18:55.251Z
paper_authors._id: 63787b13500186f250ba377c
paper_authors.avatarUrl: /avatars/96bb6051662109e9cd25e6df7d738f17.svg
paper_authors.fullname: yangsui
paper_authors.type: user
paper_authors.user: yangsui
paper_authors._id: 6836a5a9bec1d6dbb3e10459
paper_authors.name: Huan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T10:11:24.697Z
paper_authors._id: 62b624f3b52bef716e248fd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
paper_authors.fullname: Huan Wang
paper_authors.type: user
paper_authors.user: Huan-WhoRegisteredMyName
paper_title: HoliTom: Holistic Token Merging for Fast Video Large Language Models
paper_summary: Video large language models (video LLMs) excel at video comprehension but
  face significant computational inefficiency due to redundant video tokens.
  Existing token pruning methods offer solutions. However, approaches operating
  within the LLM (inner-LLM pruning), such as FastV, incur intrinsic
  computational overhead in shallow layers. In contrast, methods performing token
  pruning before the LLM (outer-LLM pruning) primarily address spatial redundancy
  within individual frames or limited temporal windows, neglecting the crucial
  global temporal dynamics and correlations across longer video sequences. This
  leads to sub-optimal spatio-temporal reduction and does not leverage video
  compressibility fully. Crucially, the synergistic potential and mutual
  influence of combining these strategies remain unexplored. To further reduce
  redundancy, we introduce HoliTom, a novel training-free holistic token merging
  framework. HoliTom employs outer-LLM pruning through global redundancy-aware
  temporal segmentation, followed by spatial-temporal merging to reduce visual
  tokens by over 90%, significantly alleviating the LLM's computational burden.
  Complementing this, we introduce a robust inner-LLM token similarity-based
  merging approach, designed for superior performance and compatibility with
  outer-LLM pruning. Evaluations demonstrate our method's promising
  efficiency-performance trade-off on LLaVA-OneVision-7B, reducing computational
  costs to 6.9% of FLOPs while maintaining 99.1% of the original performance.
  Furthermore, we achieve a 2.28x reduction in Time-To-First-Token (TTFT) and a
  1.32x acceleration in decoding throughput, highlighting the practical benefits
  of our integrated pruning approach for efficient video LLMs inference.
paper_ai_summary: HoliTom combines outer-LLM pruning through global temporal segmentation with inner-LLM token similarity-based merging to significantly reduce computational inefficiency in video LLMs without sacrificing performance.
paper_ai_keywords: video LLMs video tokens FastV inner-LLM pruning outer-LLM pruning global redundancy-aware temporal segmentation spatial-temporal merging HoliTom token similarity-based merging LLaVA-OneVision-7B Time-To-First-Token (TTFT) decoding throughput