PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid
  Visual Redundancy Reduction
In large vision-language models (LVLMs), images serve as inputs that carry a
wealth of information. As the idiom "A picture is worth a thousand words"
implies, representing a single image in current LVLMs can require hundreds or
even thousands of tokens. This results in significant computational costs,
which grow quadratically as input image resolution increases, thereby severely
impacting the efficiency of both training and inference. Previous approaches
have attempted to reduce the number of image tokens either before or within the
early layers of LVLMs. However, these strategies inevitably result in the loss
of crucial image information, ultimately diminishing model performance. To
address this challenge, we conduct an empirical study revealing that all visual
tokens are necessary for LVLMs in the shallow layers, and token redundancy
progressively increases in the deeper layers of the model. To this end, we
propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost
their efficiency in both training and inference with neglectable performance
loss. Specifically, we partition the LVLM into several stages and drop part of
the image tokens at the end of each stage with a pre-defined ratio, creating
pyramid-like visual tokens across model layers. The dropping is based on a
lightweight similarity calculation with a negligible time overhead. Extensive
experiments demonstrate that PyramidDrop can achieve a 40% training time and
55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance.
Besides, the PyramidDrop could also serve as a plug-and-play strategy for
inference acceleration without training, with better performance and lower
inference cost than counterparts. We hope that the insights and approach
introduced by PyramidDrop will inspire future research to further investigate
the role of image tokens in LVLMs.
6718923e86e0b4dce87da71f
Long Xing
6718923e86e0b4dce87da720
Qidong Huang
claimed_verified
2024-10-23T15:06:53.346Z
656f1b21b075b63c90ba02ee
/avatars/d6856815ef06261394178161e4d511b4.svg
Huang Qidong
user
shikiw
6718923e86e0b4dce87da721
Xiaoyi Dong
6718923e86e0b4dce87da722
Jiajie Lu
6718923e86e0b4dce87da723
Pan Zhang
6718923e86e0b4dce87da724
Yuhang Zang
claimed_verified
2024-10-23T07:33:08.102Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
6718923e86e0b4dce87da725
Yuhang Cao
admin_assigned
2024-10-23T07:46:39.829Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
6718923e86e0b4dce87da726
Conghui He
admin_assigned
2024-10-23T07:46:45.998Z
63f9fca8d4349b157a109eec
/avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
Conghui He
user
conghui
6718923e86e0b4dce87da727
Jiaqi Wang
admin_assigned
2024-10-23T07:47:04.518Z
661a4169bcd78151e509fd86
/avatars/2ebe19111f4c77037db875016dce017e.svg
jiaqi wang
user
jiaqiwang-rex
6718923e86e0b4dce87da728
Feng Wu
6718923e86e0b4dce87da729
Dahua Lin
admin_assigned
2024-10-23T07:47:15.494Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid
  Visual Redundancy Reduction
In large vision-language models (LVLMs), images serve as inputs that carry a
wealth of information. As the idiom "A picture is worth a thousand words"
implies, representing a single image in current LVLMs can require hundreds or
even thousands of tokens. This results in significant computational costs,
which grow quadratically as input image resolution increases, thereby severely
impacting the efficiency of both training and inference. Previous approaches
have attempted to reduce the number of image tokens either before or within the
early layers of LVLMs. However, these strategies inevitably result in the loss
of crucial image information, ultimately diminishing model performance. To
address this challenge, we conduct an empirical study revealing that all visual
tokens are necessary for LVLMs in the shallow layers, and token redundancy
progressively increases in the deeper layers of the model. To this end, we
propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost
their efficiency in both training and inference with neglectable performance
loss. Specifically, we partition the LVLM into several stages and drop part of
the image tokens at the end of each stage with a pre-defined ratio, creating
pyramid-like visual tokens across model layers. The dropping is based on a
lightweight similarity calculation with a negligible time overhead. Extensive
experiments demonstrate that PyramidDrop can achieve a 40% training time and
55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance.
Besides, the PyramidDrop could also serve as a plug-and-play strategy for
inference acceleration without training, with better performance and lower
inference cost than counterparts. We hope that the insights and approach
introduced by PyramidDrop will inspire future research to further investigate
the role of image tokens in LVLMs.
PyramidDrop, a novel strategy for reducing visual token redundancy in large vision-language models, enhances training and inference efficiency with minimal performance loss.
large vision-language models (LVLMs) image tokens computational costs PyramidDrop visual redundancy reduction token redundancy partitioning similarity calculation LLaVA-NeXT inference acceleration