title: ShortV: Efficient Multimodal Large Language Models by Freezing Visual
    Tokens in Ineffective Layers
summary: Multimodal Large Language Models (MLLMs) suffer from high computational costs
  due to their massive size and the large number of visual tokens. In this paper,
  we investigate layer-wise redundancy in MLLMs by introducing a novel metric,
  Layer Contribution (LC), which quantifies the impact of a layer's
  transformations on visual and text tokens, respectively. The calculation of LC
  involves measuring the divergence in model output that results from removing
  the layer's transformations on the specified tokens. Our pilot experiment
  reveals that many layers of MLLMs exhibit minimal contribution during the
  processing of visual tokens. Motivated by this observation, we propose ShortV,
  a training-free method that leverages LC to identify ineffective layers, and
  freezes visual token updates in these layers. Experiments show that ShortV can
  freeze visual token in approximately 60\% of the MLLM layers, thereby
  dramatically reducing computational costs related to updating visual tokens.
  For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while
  maintaining superior performance. The code will be publicly available at
  https://github.com/icip-cas/ShortV
paper_authors._id: 67ef72898667ee5c99026d16
paper_authors.name: Qianhao Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:23:28.149Z
paper_authors._id: 67014d33126f9ab39fc52481
paper_authors.avatarUrl: /avatars/60d1f791e7f3201ce1aef72e9216ff78.svg
paper_authors.fullname: Qianhao Yuan
paper_authors.type: user
paper_authors.user: yuanqianhao
paper_authors._id: 67ef72898667ee5c99026d17
paper_authors.name: Qingyu Zhang
paper_authors._id: 67ef72898667ee5c99026d18
paper_authors.name: Yanjiang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-10T06:47:11.104Z
paper_authors._id: 643b62ac065961b2252abb7a
paper_authors.avatarUrl: /avatars/c7fb4d11f0d795a52bdc771c04a69a20.svg
paper_authors.fullname: zuijiang
paper_authors.type: user
paper_authors.user: zuijiang
paper_authors._id: 67ef72898667ee5c99026d19
paper_authors.name: Jiawei Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:04.134Z
paper_authors._id: 654c7fbe6b51714c2a6ff590
paper_authors.avatarUrl: /avatars/db217415c56730872b9a807f3afb4e5b.svg
paper_authors.fullname: Jiawei Chen
paper_authors.type: user
paper_authors.user: jiawei-ucas
paper_authors._id: 67ef72898667ee5c99026d1a
paper_authors.name: Yaojie Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:24:09.511Z
paper_authors._id: 6216496a9b34d2fb49144599
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6216496a9b34d2fb49144599/41CKA_h1Ffj3RzVabSAkm.jpeg
paper_authors.fullname: Yaojie Lu
paper_authors.type: user
paper_authors.user: luyaojie
paper_authors._id: 67ef72898667ee5c99026d1b
paper_authors.name: Hongyu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:24:15.188Z
paper_authors._id: 6711c702f858a456b4b9f3a4
paper_authors.avatarUrl: /avatars/178e9567c3111ab22717c3c0dd003a6a.svg
paper_authors.fullname: Hongyu  Lin
paper_authors.type: user
paper_authors.user: sanmusunrise
paper_authors._id: 67ef72898667ee5c99026d1c
paper_authors.name: Jia Zheng
paper_authors._id: 67ef72898667ee5c99026d1d
paper_authors.name: Xianpei Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:24:23.046Z
paper_authors._id: 65e99a77e71555ed193609cf
paper_authors.avatarUrl: /avatars/38ceb127883944677665da967d17dd18.svg
paper_authors.fullname: Xianpei Han
paper_authors.type: user
paper_authors.user: xphan
paper_authors._id: 67ef72898667ee5c99026d1e
paper_authors.name: Le Sun
paper_title: ShortV: Efficient Multimodal Large Language Models by Freezing Visual
    Tokens in Ineffective Layers
paper_summary: Multimodal Large Language Models (MLLMs) suffer from high computational costs
  due to their massive size and the large number of visual tokens. In this paper,
  we investigate layer-wise redundancy in MLLMs by introducing a novel metric,
  Layer Contribution (LC), which quantifies the impact of a layer's
  transformations on visual and text tokens, respectively. The calculation of LC
  involves measuring the divergence in model output that results from removing
  the layer's transformations on the specified tokens. Our pilot experiment
  reveals that many layers of MLLMs exhibit minimal contribution during the
  processing of visual tokens. Motivated by this observation, we propose ShortV,
  a training-free method that leverages LC to identify ineffective layers, and
  freezes visual token updates in these layers. Experiments show that ShortV can
  freeze visual token in approximately 60\% of the MLLM layers, thereby
  dramatically reducing computational costs related to updating visual tokens.
  For example, it achieves a 50\% reduction in FLOPs on LLaVA-NeXT-13B while
  maintaining superior performance. The code will be publicly available at
  https://github.com/icip-cas/ShortV
paper_ai_summary: ShortV identifies and freezes ineffective layers in MLLMs to reduce computational costs without significant performance loss.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) Layer Contribution (LC) visual tokens text tokens layer-wise redundancy FLOPs