FastVLM: Efficient Vision Encoding for Vision Language Models
Scaling the input image resolution is essential for enhancing the performance
of Vision Language Models (VLMs), particularly in text-rich image understanding
tasks. However, popular visual encoders such as ViTs become inefficient at high
resolutions due to the large number of tokens and high encoding latency caused
by stacked self-attention layers. At different operational resolutions, the
vision encoder of a VLM can be optimized along two axes: reducing encoding
latency and minimizing the number of visual tokens passed to the LLM, thereby
lowering overall latency. Based on a comprehensive efficiency analysis of the
interplay between image resolution, vision latency, token count, and LLM size,
we introduce FastVLM, a model that achieves an optimized trade-off between
latency, model size and accuracy. FastVLM incorporates FastViTHD, a novel
hybrid vision encoder designed to output fewer tokens and significantly reduce
encoding time for high-resolution images. Unlike previous methods, FastVLM
achieves the optimal balance between visual token count and image resolution
solely by scaling the input image, eliminating the need for additional token
pruning and simplifying the model design. In the LLaVA-1.5 setup, FastVLM
achieves 3.2times improvement in time-to-first-token (TTFT) while
maintaining similar performance on VLM benchmarks compared to prior works.
Compared to LLaVa-OneVision at the highest resolution (1152times1152),
FastVLM achieves comparable performance on key benchmarks like SeedBench and
MMMU, using the same 0.5B LLM, but with 85times faster TTFT and a vision
encoder that is 3.4times smaller.
6764433e98b6fdd712b6f62c
Pavan Kumar Anasosalu Vasu
claimed_verified
2024-12-30T19:36:35.863Z
65e000cce1823ce0df17a3d4
/avatars/b50c2d2c1f3ad1114867e199fd578a29.svg
Pavan Kumar Vasu
user
pavankumarvasu
6764433e98b6fdd712b6f62d
Fartash Faghri
admin_assigned
2024-12-20T09:58:10.778Z
65f8945a3ba58880f9794a11
/avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
Fartash Faghri
user
fartashf
6764433e98b6fdd712b6f62e
Chun-Liang Li
admin_assigned
2024-12-20T09:58:04.623Z
65de6ef104b9352580ce23bc
/avatars/aace4a7c8ee9a591f1103a2b38f9b9ff.svg
Chun-Liang Li
user
chunliangli
6764433e98b6fdd712b6f62f
Cem Koc
admin_assigned
2024-12-20T09:57:57.333Z
62662bf4ab93b672352de36f
/avatars/58d44088f94cb2b5dbba099d67d8df84.svg
Cem Koc
user
cemkoc
6764433e98b6fdd712b6f630
Nate True
admin_assigned
2024-12-20T09:57:35.922Z
6328b013ce50f125374c1f56
/avatars/e9d01d9e50f7cde556bf3bc37856efff.svg
Nate True
user
natetrue
6764433e98b6fdd712b6f631
Albert Antony
admin_assigned
2024-12-20T09:57:29.611Z
654c9ac26f2ec14d77139062
/avatars/8a2f42591862bddf758995256a01eb38.svg
Albert Antony
user
albertantony
6764433e98b6fdd712b6f632
Gokul Santhanam
6764433e98b6fdd712b6f633
James Gabriel
6764433e98b6fdd712b6f634
Peter Grasch
admin_assigned
2024-12-20T09:57:03.748Z
6649e0b6f50d471119285ab8
/avatars/1c93d0e15f2b68281a1a2ea052450a45.svg
Peter Grasch
user
petergrasch
6764433e98b6fdd712b6f635
Oncel Tuzel
6764433e98b6fdd712b6f636
Hadi Pouransari
admin_assigned
2024-12-20T09:56:54.004Z
653a8e65c75dc136bfb5b0f8
/avatars/ef252794236ce0fe2debf12773c95bb2.svg
Hadi Pouransari
user
hpouransari
FastVLM: Efficient Vision Encoding for Vision Language Models
Scaling the input image resolution is essential for enhancing the performance
of Vision Language Models (VLMs), particularly in text-rich image understanding
tasks. However, popular visual encoders such as ViTs become inefficient at high
resolutions due to the large number of tokens and high encoding latency caused
by stacked self-attention layers. At different operational resolutions, the
vision encoder of a VLM can be optimized along two axes: reducing encoding
latency and minimizing the number of visual tokens passed to the LLM, thereby
lowering overall latency. Based on a comprehensive efficiency analysis of the
interplay between image resolution, vision latency, token count, and LLM size,
we introduce FastVLM, a model that achieves an optimized trade-off between
latency, model size and accuracy. FastVLM incorporates FastViTHD, a novel
hybrid vision encoder designed to output fewer tokens and significantly reduce
encoding time for high-resolution images. Unlike previous methods, FastVLM
achieves the optimal balance between visual token count and image resolution
solely by scaling the input image, eliminating the need for additional token
pruning and simplifying the model design. In the LLaVA-1.5 setup, FastVLM
achieves 3.2times improvement in time-to-first-token (TTFT) while
maintaining similar performance on VLM benchmarks compared to prior works.
Compared to LLaVa-OneVision at the highest resolution (1152times1152),
FastVLM achieves comparable performance on key benchmarks like SeedBench and
MMMU, using the same 0.5B LLM, but with 85times faster TTFT and a vision
encoder that is 3.4times smaller.
FastVLM, using FastViTHD, optimizes Vision Language Models by scaling image resolution to reduce latency and token count, improving time-to-first-token and maintaining performance.
ViTs self-attention layers FastVLM FastViTHD time-to-first-token TTFT SeedBench MMMU