ShortV: Efficient Multimodal Large Language Models by Freezing Visual
  Tokens in Ineffective Layers
Multimodal Large Language Models (MLLMs) suffer from high computational costs
due to their massive size and the large number of visual tokens. In this paper,
we investigate layer-wise redundancy in MLLMs by introducing a novel metric,
Layer Contribution (LC), which quantifies the impact of a layer's
transformations on visual and text tokens, respectively. The calculation of LC
involves measuring the divergence in model output that results from removing
the layer's transformations on the specified tokens. Our pilot experiment
reveals that many layers of MLLMs exhibit minimal contribution during the
processing of visual tokens. Motivated by this observation, we propose ShortV,
a training-free method that leverages LC to identify ineffective layers, and
freezes visual token updates in these layers. Experiments show that ShortV can
freeze visual token in approximately 60\% of the MLLM layers, thereby
dramatically reducing computational costs related to updating visual tokens.
For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while
maintaining superior performance. The code will be publicly available at
https://github.com/icip-cas/ShortV
67ef72898667ee5c99026d16
Qianhao Yuan
admin_assigned
2025-04-04T07:23:28.149Z
67014d33126f9ab39fc52481
/avatars/60d1f791e7f3201ce1aef72e9216ff78.svg
Qianhao Yuan
user
yuanqianhao
67ef72898667ee5c99026d17
Qingyu Zhang
67ef72898667ee5c99026d18
Yanjiang Liu
claimed_verified
2025-04-10T06:47:11.104Z
643b62ac065961b2252abb7a
/avatars/c7fb4d11f0d795a52bdc771c04a69a20.svg
zuijiang
user
zuijiang
67ef72898667ee5c99026d19
Jiawei Chen
claimed_verified
2025-04-04T07:08:04.134Z
654c7fbe6b51714c2a6ff590
/avatars/db217415c56730872b9a807f3afb4e5b.svg
Jiawei Chen
user
jiawei-ucas
67ef72898667ee5c99026d1a
Yaojie Lu
admin_assigned
2025-04-04T07:24:09.511Z
6216496a9b34d2fb49144599
https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
Yaojie Lu
user
luyaojie
67ef72898667ee5c99026d1b
Hongyu Lin
admin_assigned
2025-04-04T07:24:15.188Z
6711c702f858a456b4b9f3a4
/avatars/178e9567c3111ab22717c3c0dd003a6a.svg
Hongyu  Lin
user
sanmusunrise
67ef72898667ee5c99026d1c
Jia Zheng
67ef72898667ee5c99026d1d
Xianpei Han
admin_assigned
2025-04-04T07:24:23.046Z
65e99a77e71555ed193609cf
/avatars/38ceb127883944677665da967d17dd18.svg
Xianpei Han
user
xphan
67ef72898667ee5c99026d1e
Le Sun
ShortV: Efficient Multimodal Large Language Models by Freezing Visual
  Tokens in Ineffective Layers
Multimodal Large Language Models (MLLMs) suffer from high computational costs
due to their massive size and the large number of visual tokens. In this paper,
we investigate layer-wise redundancy in MLLMs by introducing a novel metric,
Layer Contribution (LC), which quantifies the impact of a layer's
transformations on visual and text tokens, respectively. The calculation of LC
involves measuring the divergence in model output that results from removing
the layer's transformations on the specified tokens. Our pilot experiment
reveals that many layers of MLLMs exhibit minimal contribution during the
processing of visual tokens. Motivated by this observation, we propose ShortV,
a training-free method that leverages LC to identify ineffective layers, and
freezes visual token updates in these layers. Experiments show that ShortV can
freeze visual token in approximately 60\% of the MLLM layers, thereby
dramatically reducing computational costs related to updating visual tokens.
For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while
maintaining superior performance. The code will be publicly available at
https://github.com/icip-cas/ShortV
ShortV identifies and freezes ineffective layers in MLLMs to reduce computational costs without significant performance loss.
Multimodal Large Language Models (MLLMs) Layer Contribution (LC) visual tokens text tokens layer-wise redundancy FLOPs