title: HiRED: Attention-Guided Token Dropping for Efficient Inference of
    High-Resolution Vision-Language Models in Resource-Constrained Environments
summary: High-resolution Vision-Language Models (VLMs) have been widely used in
  multimodal tasks to enhance accuracy by preserving detailed image information.
  However, these models often generate excessive visual tokens due to encoding
  multiple partitions of the input image. Processing these excessive visual
  tokens is computationally challenging, especially in resource-constrained
  environments with commodity GPUs. To support high-resolution images while
  meeting resource constraints, we propose High-Resolution Early Dropping
  (HiRED), a token-dropping scheme that operates within a fixed token budget
  before the Large Language Model (LLM) stage. HiRED can be integrated with
  existing high-resolution VLMs in a plug-and-play manner, as it requires no
  additional training while still maintaining superior accuracy. We strategically
  use the vision encoder's attention in the initial layers to assess the visual
  content of each image partition and allocate the token budget accordingly.
  Then, using the attention in the final layer, we select the most important
  visual tokens from each partition within the allocated budget, dropping the
  rest. Empirically, when applied to LLaVA-Next-7B on NVIDIA TESLA P40 GPU, HiRED
  with a 20% token budget increases token generation throughput by 4.7, reduces
  first-token generation latency by 15 seconds, and saves 2.3 GB of GPU memory
  for a single inference.
paper_authors._id: 66c80f9ade9f2c4dad1bd5b4
paper_authors.name: Kazi Hasan Ibn Arif
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-26T07:15:19.294Z
paper_authors._id: 654af6f173416a223f5eacf5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/FZvER_MtUz_BitTVf2QfV.jpeg
paper_authors.fullname: Hasan Arif
paper_authors.type: user
paper_authors.user: hasanar1f
paper_authors._id: 66c80f9ade9f2c4dad1bd5b5
paper_authors.name: JinYi Yoon
paper_authors._id: 66c80f9ade9f2c4dad1bd5b6
paper_authors.name: Dimitrios S. Nikolopoulos
paper_authors._id: 66c80f9ade9f2c4dad1bd5b7
paper_authors.name: Hans Vandierendonck
paper_authors._id: 66c80f9ade9f2c4dad1bd5b8
paper_authors.name: Deepu John
paper_authors._id: 66c80f9ade9f2c4dad1bd5b9
paper_authors.name: Bo Ji
paper_title: HiRED: Attention-Guided Token Dropping for Efficient Inference of
    High-Resolution Vision-Language Models in Resource-Constrained Environments
paper_summary: High-resolution Vision-Language Models (VLMs) have been widely used in
  multimodal tasks to enhance accuracy by preserving detailed image information.
  However, these models often generate excessive visual tokens due to encoding
  multiple partitions of the input image. Processing these excessive visual
  tokens is computationally challenging, especially in resource-constrained
  environments with commodity GPUs. To support high-resolution images while
  meeting resource constraints, we propose High-Resolution Early Dropping
  (HiRED), a token-dropping scheme that operates within a fixed token budget
  before the Large Language Model (LLM) stage. HiRED can be integrated with
  existing high-resolution VLMs in a plug-and-play manner, as it requires no
  additional training while still maintaining superior accuracy. We strategically
  use the vision encoder's attention in the initial layers to assess the visual
  content of each image partition and allocate the token budget accordingly.
  Then, using the attention in the final layer, we select the most important
  visual tokens from each partition within the allocated budget, dropping the
  rest. Empirically, when applied to LLaVA-Next-7B on NVIDIA TESLA P40 GPU, HiRED
  with a 20% token budget increases token generation throughput by 4.7, reduces
  first-token generation latency by 15 seconds, and saves 2.3 GB of GPU memory
  for a single inference.
paper_ai_summary: High-Resolution Early Dropping (HiRED) improves computational efficiency and resource usage in high-resolution Vision-Language Models by optimizing token allocation and generation.
paper_ai_keywords: High-Resolution Vision-Language Models VLMs token-dropping scheme Large Language Model LLM vision encoder's attention token budget NVIDIA TESLA P40 GPU token generation throughput first-token generation latency GPU memory