title: VisionZip: Longer is Better but Not Necessary in Vision Language Models
summary: Recent advancements in vision-language models have enhanced performance by
  increasing the length of visual tokens, making them much longer than text
  tokens and significantly raising computational costs. However, we observe that
  the visual tokens generated by popular vision encoders, such as CLIP and
  SigLIP, contain significant redundancy. To address this, we introduce
  VisionZip, a simple yet effective method that selects a set of informative
  tokens for input to the language model, reducing visual token redundancy and
  improving efficiency while maintaining model performance. The proposed
  VisionZip can be widely applied to image and video understanding tasks and is
  well-suited for multi-turn dialogues in real-world scenarios, where previous
  methods tend to underperform. Experimental results show that VisionZip
  outperforms the previous state-of-the-art method by at least 5% performance
  gains across nearly all settings. Moreover, our method significantly enhances
  model inference speed, improving the prefilling time by 8x and enabling the
  LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while
  achieving better results. Furthermore, we analyze the causes of this redundancy
  and encourage the community to focus on extracting better visual features
  rather than merely increasing token length. Our code is available at
  https://github.com/dvlab-research/VisionZip .
paper_authors._id: 6752642e9106cbed4a680b9d
paper_authors.name: Senqiao Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:21.290Z
paper_authors._id: 6527b7280ae663e384eb8499
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
paper_authors.fullname: Senqiao Yang
paper_authors.type: user
paper_authors.user: Senqiao
paper_authors._id: 6752642e9106cbed4a680b9e
paper_authors.name: Yukang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T09:45:09.922Z
paper_authors._id: 65b31d77f32c79fea79ba4d1
paper_authors.avatarUrl: /avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
paper_authors.fullname: Yukang Chen
paper_authors.type: user
paper_authors.user: Yukang2
paper_authors._id: 6752642e9106cbed4a680b9f
paper_authors.name: Zhuotao Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T09:45:21.690Z
paper_authors._id: 64d104a37a7305c5895bd720
paper_authors.avatarUrl: /avatars/2d9eff3a2dff6d02e45ae8964fb91f27.svg
paper_authors.fullname: zt tian
paper_authors.type: user
paper_authors.user: tianzhuotao
paper_authors._id: 6752642e9106cbed4a680ba0
paper_authors.name: Chengyao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T09:45:28.672Z
paper_authors._id: 64c3257a027e6d966673712a
paper_authors.avatarUrl: /avatars/abac7539c8e90c2780ecc445536d8c2c.svg
paper_authors.fullname: Chengyao Wang
paper_authors.type: user
paper_authors.user: wangchengyao
paper_authors._id: 6752642e9106cbed4a680ba1
paper_authors.name: Jingyao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T09:45:35.074Z
paper_authors._id: 647ff779d037515be1169e35
paper_authors.avatarUrl: /avatars/741096302cbdeb3670a8f31cf2d31d5d.svg
paper_authors.fullname: Jingyao Li
paper_authors.type: user
paper_authors.user: JingyaoLi
paper_authors._id: 6752642e9106cbed4a680ba2
paper_authors.name: Bei Yu
paper_authors._id: 6752642e9106cbed4a680ba3
paper_authors.name: Jiaya Jia
paper_title: VisionZip: Longer is Better but Not Necessary in Vision Language Models
paper_summary: Recent advancements in vision-language models have enhanced performance by
  increasing the length of visual tokens, making them much longer than text
  tokens and significantly raising computational costs. However, we observe that
  the visual tokens generated by popular vision encoders, such as CLIP and
  SigLIP, contain significant redundancy. To address this, we introduce
  VisionZip, a simple yet effective method that selects a set of informative
  tokens for input to the language model, reducing visual token redundancy and
  improving efficiency while maintaining model performance. The proposed
  VisionZip can be widely applied to image and video understanding tasks and is
  well-suited for multi-turn dialogues in real-world scenarios, where previous
  methods tend to underperform. Experimental results show that VisionZip
  outperforms the previous state-of-the-art method by at least 5% performance
  gains across nearly all settings. Moreover, our method significantly enhances
  model inference speed, improving the prefilling time by 8x and enabling the
  LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while
  achieving better results. Furthermore, we analyze the causes of this redundancy
  and encourage the community to focus on extracting better visual features
  rather than merely increasing token length. Our code is available at
  https://github.com/dvlab-research/VisionZip .
paper_ai_summary: VisionZip reduces visual token redundancy and enhances efficiency in vision-language models, improving performance and inference speed across various tasks.
paper_ai_keywords: vision-language models visual tokens text tokens CLIP SigLIP VisionZip image and video understanding tasks multi-turn dialogues prefilling time LLaVA-Next