HiRED: Attention-Guided Token Dropping for Efficient Inference of
  High-Resolution Vision-Language Models in Resource-Constrained Environments
High-resolution Vision-Language Models (VLMs) have been widely used in
multimodal tasks to enhance accuracy by preserving detailed image information.
However, these models often generate excessive visual tokens due to encoding
multiple partitions of the input image. Processing these excessive visual
tokens is computationally challenging, especially in resource-constrained
environments with commodity GPUs. To support high-resolution images while
meeting resource constraints, we propose High-Resolution Early Dropping
(HiRED), a token-dropping scheme that operates within a fixed token budget
before the Large Language Model (LLM) stage. HiRED can be integrated with
existing high-resolution VLMs in a plug-and-play manner, as it requires no
additional training while still maintaining superior accuracy. We strategically
use the vision encoder's attention in the initial layers to assess the visual
content of each image partition and allocate the token budget accordingly.
Then, using the attention in the final layer, we select the most important
visual tokens from each partition within the allocated budget, dropping the
rest. Empirically, when applied to LLaVA-Next-7B on NVIDIA TESLA P40 GPU, HiRED
with a 20% token budget increases token generation throughput by 4.7, reduces
first-token generation latency by 15 seconds, and saves 2.3 GB of GPU memory
for a single inference.
66c80f9ade9f2c4dad1bd5b4
Kazi Hasan Ibn Arif
claimed_verified
2024-08-26T07:15:19.294Z
654af6f173416a223f5eacf5
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/FZvER_MtUz_BitTVf2QfV.jpeg
Hasan Arif
user
hasanar1f
66c80f9ade9f2c4dad1bd5b5
JinYi Yoon
66c80f9ade9f2c4dad1bd5b6
Dimitrios S. Nikolopoulos
66c80f9ade9f2c4dad1bd5b7
Hans Vandierendonck
66c80f9ade9f2c4dad1bd5b8
Deepu John
66c80f9ade9f2c4dad1bd5b9
Bo Ji
HiRED: Attention-Guided Token Dropping for Efficient Inference of
  High-Resolution Vision-Language Models in Resource-Constrained Environments
High-resolution Vision-Language Models (VLMs) have been widely used in
multimodal tasks to enhance accuracy by preserving detailed image information.
However, these models often generate excessive visual tokens due to encoding
multiple partitions of the input image. Processing these excessive visual
tokens is computationally challenging, especially in resource-constrained
environments with commodity GPUs. To support high-resolution images while
meeting resource constraints, we propose High-Resolution Early Dropping
(HiRED), a token-dropping scheme that operates within a fixed token budget
before the Large Language Model (LLM) stage. HiRED can be integrated with
existing high-resolution VLMs in a plug-and-play manner, as it requires no
additional training while still maintaining superior accuracy. We strategically
use the vision encoder's attention in the initial layers to assess the visual
content of each image partition and allocate the token budget accordingly.
Then, using the attention in the final layer, we select the most important
visual tokens from each partition within the allocated budget, dropping the
rest. Empirically, when applied to LLaVA-Next-7B on NVIDIA TESLA P40 GPU, HiRED
with a 20% token budget increases token generation throughput by 4.7, reduces
first-token generation latency by 15 seconds, and saves 2.3 GB of GPU memory
for a single inference.
High-Resolution Early Dropping (HiRED) improves computational efficiency and resource usage in high-resolution Vision-Language Models by optimizing token allocation and generation.
High-Resolution Vision-Language Models VLMs token-dropping scheme Large Language Model LLM vision encoder's attention token budget NVIDIA TESLA P40 GPU token generation throughput first-token generation latency GPU memory