title: PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid
    Visual Redundancy Reduction
summary: In large vision-language models (LVLMs), images serve as inputs that carry a
  wealth of information. As the idiom "A picture is worth a thousand words"
  implies, representing a single image in current LVLMs can require hundreds or
  even thousands of tokens. This results in significant computational costs,
  which grow quadratically as input image resolution increases, thereby severely
  impacting the efficiency of both training and inference. Previous approaches
  have attempted to reduce the number of image tokens either before or within the
  early layers of LVLMs. However, these strategies inevitably result in the loss
  of crucial image information, ultimately diminishing model performance. To
  address this challenge, we conduct an empirical study revealing that all visual
  tokens are necessary for LVLMs in the shallow layers, and token redundancy
  progressively increases in the deeper layers of the model. To this end, we
  propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost
  their efficiency in both training and inference with neglectable performance
  loss. Specifically, we partition the LVLM into several stages and drop part of
  the image tokens at the end of each stage with a pre-defined ratio, creating
  pyramid-like visual tokens across model layers. The dropping is based on a
  lightweight similarity calculation with a negligible time overhead. Extensive
  experiments demonstrate that PyramidDrop can achieve a 40% training time and
  55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance.
  Besides, the PyramidDrop could also serve as a plug-and-play strategy for
  inference acceleration without training, with better performance and lower
  inference cost than counterparts. We hope that the insights and approach
  introduced by PyramidDrop will inspire future research to further investigate
  the role of image tokens in LVLMs.
paper_authors._id: 6718923e86e0b4dce87da71f
paper_authors.name: Long Xing
paper_authors._id: 6718923e86e0b4dce87da720
paper_authors.name: Qidong Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T15:06:53.346Z
paper_authors._id: 656f1b21b075b63c90ba02ee
paper_authors.avatarUrl: /avatars/d6856815ef06261394178161e4d511b4.svg
paper_authors.fullname: Huang Qidong
paper_authors.type: user
paper_authors.user: shikiw
paper_authors._id: 6718923e86e0b4dce87da721
paper_authors.name: Xiaoyi Dong
paper_authors._id: 6718923e86e0b4dce87da722
paper_authors.name: Jiajie Lu
paper_authors._id: 6718923e86e0b4dce87da723
paper_authors.name: Pan Zhang
paper_authors._id: 6718923e86e0b4dce87da724
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T07:33:08.102Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 6718923e86e0b4dce87da725
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T07:46:39.829Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 6718923e86e0b4dce87da726
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T07:46:45.998Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 6718923e86e0b4dce87da727
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T07:47:04.518Z
paper_authors._id: 661a4169bcd78151e509fd86
paper_authors.avatarUrl: /avatars/2ebe19111f4c77037db875016dce017e.svg
paper_authors.fullname: jiaqi wang
paper_authors.type: user
paper_authors.user: jiaqiwang-rex
paper_authors._id: 6718923e86e0b4dce87da728
paper_authors.name: Feng Wu
paper_authors._id: 6718923e86e0b4dce87da729
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-23T07:47:15.494Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_title: PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid
    Visual Redundancy Reduction
paper_summary: In large vision-language models (LVLMs), images serve as inputs that carry a
  wealth of information. As the idiom "A picture is worth a thousand words"
  implies, representing a single image in current LVLMs can require hundreds or
  even thousands of tokens. This results in significant computational costs,
  which grow quadratically as input image resolution increases, thereby severely
  impacting the efficiency of both training and inference. Previous approaches
  have attempted to reduce the number of image tokens either before or within the
  early layers of LVLMs. However, these strategies inevitably result in the loss
  of crucial image information, ultimately diminishing model performance. To
  address this challenge, we conduct an empirical study revealing that all visual
  tokens are necessary for LVLMs in the shallow layers, and token redundancy
  progressively increases in the deeper layers of the model. To this end, we
  propose PyramidDrop, a visual redundancy reduction strategy for LVLMs to boost
  their efficiency in both training and inference with neglectable performance
  loss. Specifically, we partition the LVLM into several stages and drop part of
  the image tokens at the end of each stage with a pre-defined ratio, creating
  pyramid-like visual tokens across model layers. The dropping is based on a
  lightweight similarity calculation with a negligible time overhead. Extensive
  experiments demonstrate that PyramidDrop can achieve a 40% training time and
  55% inference FLOPs acceleration of LLaVA-NeXT with comparable performance.
  Besides, the PyramidDrop could also serve as a plug-and-play strategy for
  inference acceleration without training, with better performance and lower
  inference cost than counterparts. We hope that the insights and approach
  introduced by PyramidDrop will inspire future research to further investigate
  the role of image tokens in LVLMs.
paper_ai_keywords: vision-language models (LVLMs) image tokens token redundancy PyramidDrop visual redundancy reduction strategy similarity calculation