A Glimpse to Compress: Dynamic Visual Token Pruning for Large
  Vision-Language Models
Visual token compression is critical for Large Vision-Language Models (LVLMs)
to efficiently process high-resolution inputs. Existing methods that typically
adopt fixed compression ratios cannot adapt to scenes of varying complexity,
often causing imprecise pruning that discards informative visual tokens and
results in degraded model performance. To address this issue, we introduce a
dynamic pruning framework, GlimpsePrune, inspired by human cognition. It takes
a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single
forward pass before answer generation. This approach prunes 92.6% of visual
tokens while on average fully retaining the baseline performance on free-form
VQA tasks. The reduced computational cost also enables more effective
fine-tuning: an enhanced GlimpsePrune+ achieves 110% of the baseline
performance while maintaining a similarly high pruning rate. Our work paves a
new way for building more powerful and efficient LVLMs.
689186aef01a094725f83537
Quan-Sheng Zeng
689186aef01a094725f83538
Yunheng Li
689186aef01a094725f83539
Qilong Wang
689186aef01a094725f8353a
Peng-Tao Jiang
689186aef01a094725f8353b
Zuxuan Wu
689186aef01a094725f8353c
Ming-Ming Cheng
689186aef01a094725f8353d
Qibin Hou
A Glimpse to Compress: Dynamic Visual Token Pruning for Large
  Vision-Language Models
Visual token compression is critical for Large Vision-Language Models (LVLMs)
to efficiently process high-resolution inputs. Existing methods that typically
adopt fixed compression ratios cannot adapt to scenes of varying complexity,
often causing imprecise pruning that discards informative visual tokens and
results in degraded model performance. To address this issue, we introduce a
dynamic pruning framework, GlimpsePrune, inspired by human cognition. It takes
a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single
forward pass before answer generation. This approach prunes 92.6% of visual
tokens while on average fully retaining the baseline performance on free-form
VQA tasks. The reduced computational cost also enables more effective
fine-tuning: an enhanced GlimpsePrune+ achieves 110% of the baseline
performance while maintaining a similarly high pruning rate. Our work paves a
new way for building more powerful and efficient LVLMs.
A dynamic pruning framework, GlimpsePrune, improves efficiency in Large Vision-Language Models by adaptively removing irrelevant visual tokens without degrading performance.
Visual token compression Large Vision-Language Models dynamic pruning GlimpsePrune free-form VQA tasks parameter-efficient fine-tuning