title: An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference
    Acceleration for Large Vision-Language Models
summary: In this study, we identify the inefficient attention phenomena in Large
  Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5,
  QwenVL-Chat and Video-LLaVA. We find out that the attention computation over
  visual tokens is of extreme inefficiency in the deep layers of popular LVLMs,
  suggesting a need for a sparser approach compared to textual data handling. To
  this end, we introduce FastV, a versatile plug-and-play method designed to
  optimize computational efficiency by learning adaptive attention patterns in
  early layers and pruning visual tokens in subsequent ones. Our evaluations
  demonstrate FastV's ability to dramatically reduce computational costs (e.g., a
  45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a
  wide range of image and video understanding tasks. The computational efficiency
  and performance trade-off of FastV are highly customizable and
  pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve
  a lower budget than that of a 7B-parameter model, while still maintaining
  superior performance. We believe FastV has practical values for deployment of
  LVLMs in edge devices and commercial models. Code is released at
  https://github.com/pkunlp-icler/FastV.
paper_authors._id: 65efce192392e7d5fb0bd36e
paper_authors.name: Liang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-12T12:06:02.476Z
paper_authors._id: 61b0a4ce1b3d95b3d1ed9251
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: leonardPKU
paper_authors._id: 65efce192392e7d5fb0bd36f
paper_authors.name: Haozhe Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T08:43:42.965Z
paper_authors._id: 63ff09a02098b9ad105a09f6
paper_authors.avatarUrl: /avatars/4409ca5d320050cf4c3df05962c7ff58.svg
paper_authors.fullname: Hans Zhao
paper_authors.type: user
paper_authors.user: BleachNick
paper_authors._id: 65efce192392e7d5fb0bd370
paper_authors.name: Tianyu Liu
paper_authors._id: 65efce192392e7d5fb0bd371
paper_authors.name: Shuai Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:15:13.985Z
paper_authors._id: 63451cf0a05b51f7ded25505
paper_authors.avatarUrl: /avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
paper_authors.fullname: shuai bai
paper_authors.type: user
paper_authors.user: bluelike
paper_authors._id: 65efce192392e7d5fb0bd372
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:15:20.851Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 65efce192392e7d5fb0bd373
paper_authors.name: Chang Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-12T12:15:34.914Z
paper_authors._id: 622892774323cef93a956a4a
paper_authors.avatarUrl: /avatars/e57ef5c3b0c4289988ccd42f14e54336.svg
paper_authors.fullname: chang zhou
paper_authors.type: user
paper_authors.user: jiemizc
paper_authors._id: 65efce192392e7d5fb0bd374
paper_authors.name: Baobao Chang
paper_title: An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference
    Acceleration for Large Vision-Language Models
paper_summary: In this study, we identify the inefficient attention phenomena in Large
  Vision-Language Models (LVLMs), notably within prominent models like LLaVA-1.5,
  QwenVL-Chat and Video-LLaVA. We find out that the attention computation over
  visual tokens is of extreme inefficiency in the deep layers of popular LVLMs,
  suggesting a need for a sparser approach compared to textual data handling. To
  this end, we introduce FastV, a versatile plug-and-play method designed to
  optimize computational efficiency by learning adaptive attention patterns in
  early layers and pruning visual tokens in subsequent ones. Our evaluations
  demonstrate FastV's ability to dramatically reduce computational costs (e.g., a
  45 reduction in FLOPs for LLaVA-1.5-13B) without sacrificing performance in a
  wide range of image and video understanding tasks. The computational efficiency
  and performance trade-off of FastV are highly customizable and
  pareto-efficient. It can compress the FLOPs of a 13B-parameter model to achieve
  a lower budget than that of a 7B-parameter model, while still maintaining
  superior performance. We believe FastV has practical values for deployment of
  LVLMs in edge devices and commercial models. Code is released at
  https://github.com/pkunlp-icler/FastV.
paper_ai_keywords: Large Vision-Language Models (LVLMs) LLaVA-1.5 QwenVL-Chat Video-LLaVA attention phenomena attention computation visual tokens sparser approach FastV plug-and-play method adaptive attention patterns computational efficiency pruning FLOPs (floating-point operations) state-of-the-art image understanding video understanding pareto-efficient deployment commercial models