title: Feather the Throttle: Revisiting Visual Token Pruning for
    Vision-Language Model Acceleration
summary: Recent works on accelerating Vision-Language Models show that strong
  performance can be maintained across a variety of vision-language tasks despite
  highly compressing visual information. In this work, we examine the popular
  acceleration approach of early pruning of visual tokens inside the language
  model and find that its strong performance across many tasks is not due to an
  exceptional ability to compress visual information, but rather the benchmarks'
  limited ability to assess fine-grained visual capabilities. Namely, we
  demonstrate a core issue with the acceleration approach where most tokens
  towards the top of the image are pruned away. Yet, this issue is only reflected
  in performance for a small subset of tasks such as localization. For the other
  evaluated tasks, strong performance is maintained with the flawed pruning
  strategy. Noting the limited visual capabilities of the studied acceleration
  technique, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble
  cRiteria), a straightforward approach that (1) resolves the identified issue
  with early-layer pruning, (2) incorporates uniform sampling to ensure coverage
  across all image regions, and (3) applies pruning in two stages to allow the
  criteria to become more effective at a later layer while still achieving
  significant speedup through early-layer pruning. With comparable computational
  savings, we find that FEATHER has more than 5times performance improvement
  on the vision-centric localization benchmarks compared to the original
  acceleration approach.
paper_authors._id: 6762382699046e8f14ef8319
paper_authors.name: Mark Endo
paper_authors._id: 6762382699046e8f14ef831a
paper_authors.name: Xiaohan Wang
paper_authors._id: 6762382699046e8f14ef831b
paper_authors.name: Serena Yeung-Levy
paper_title: Feather the Throttle: Revisiting Visual Token Pruning for
    Vision-Language Model Acceleration
paper_summary: Recent works on accelerating Vision-Language Models show that strong
  performance can be maintained across a variety of vision-language tasks despite
  highly compressing visual information. In this work, we examine the popular
  acceleration approach of early pruning of visual tokens inside the language
  model and find that its strong performance across many tasks is not due to an
  exceptional ability to compress visual information, but rather the benchmarks'
  limited ability to assess fine-grained visual capabilities. Namely, we
  demonstrate a core issue with the acceleration approach where most tokens
  towards the top of the image are pruned away. Yet, this issue is only reflected
  in performance for a small subset of tasks such as localization. For the other
  evaluated tasks, strong performance is maintained with the flawed pruning
  strategy. Noting the limited visual capabilities of the studied acceleration
  technique, we propose FEATHER (Fast and Effective Acceleration wiTH Ensemble
  cRiteria), a straightforward approach that (1) resolves the identified issue
  with early-layer pruning, (2) incorporates uniform sampling to ensure coverage
  across all image regions, and (3) applies pruning in two stages to allow the
  criteria to become more effective at a later layer while still achieving
  significant speedup through early-layer pruning. With comparable computational
  savings, we find that FEATHER has more than 5times performance improvement
  on the vision-centric localization benchmarks compared to the original
  acceleration approach.
paper_ai_summary: FEATHER, a novel acceleration technique for Vision-Language Models, improves localization performance significantly by addressing flaws in early-layer pruning and incorporating uniform sampling.
paper_ai_keywords: acceleration Vision-Language Models early pruning visual tokens localization FEATHER uniform sampling