title: FastVLM: Efficient Vision Encoding for Vision Language Models
summary: Scaling the input image resolution is essential for enhancing the performance
  of Vision Language Models (VLMs), particularly in text-rich image understanding
  tasks. However, popular visual encoders such as ViTs become inefficient at high
  resolutions due to the large number of tokens and high encoding latency caused
  by stacked self-attention layers. At different operational resolutions, the
  vision encoder of a VLM can be optimized along two axes: reducing encoding
  latency and minimizing the number of visual tokens passed to the LLM, thereby
  lowering overall latency. Based on a comprehensive efficiency analysis of the
  interplay between image resolution, vision latency, token count, and LLM size,
  we introduce FastVLM, a model that achieves an optimized trade-off between
  latency, model size and accuracy. FastVLM incorporates FastViTHD, a novel
  hybrid vision encoder designed to output fewer tokens and significantly reduce
  encoding time for high-resolution images. Unlike previous methods, FastVLM
  achieves the optimal balance between visual token count and image resolution
  solely by scaling the input image, eliminating the need for additional token
  pruning and simplifying the model design. In the LLaVA-1.5 setup, FastVLM
  achieves 3.2times improvement in time-to-first-token (TTFT) while
  maintaining similar performance on VLM benchmarks compared to prior works.
  Compared to LLaVa-OneVision at the highest resolution (1152times1152),
  FastVLM achieves comparable performance on key benchmarks like SeedBench and
  MMMU, using the same 0.5B LLM, but with 85times faster TTFT and a vision
  encoder that is 3.4times smaller.
paper_authors._id: 6764433e98b6fdd712b6f62c
paper_authors.name: Pavan Kumar Anasosalu Vasu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:36:35.863Z
paper_authors._id: 65e000cce1823ce0df17a3d4
paper_authors.avatarUrl: /avatars/b50c2d2c1f3ad1114867e199fd578a29.svg
paper_authors.fullname: Pavan Kumar Vasu
paper_authors.type: user
paper_authors.user: pavankumarvasu
paper_authors._id: 6764433e98b6fdd712b6f62d
paper_authors.name: Fartash Faghri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:58:10.778Z
paper_authors._id: 65f8945a3ba58880f9794a11
paper_authors.avatarUrl: /avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
paper_authors.fullname: Fartash Faghri
paper_authors.type: user
paper_authors.user: fartashf
paper_authors._id: 6764433e98b6fdd712b6f62e
paper_authors.name: Chun-Liang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:58:04.623Z
paper_authors._id: 65de6ef104b9352580ce23bc
paper_authors.avatarUrl: /avatars/aace4a7c8ee9a591f1103a2b38f9b9ff.svg
paper_authors.fullname: Chun-Liang Li
paper_authors.type: user
paper_authors.user: chunliangli
paper_authors._id: 6764433e98b6fdd712b6f62f
paper_authors.name: Cem Koc
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:57:57.333Z
paper_authors._id: 62662bf4ab93b672352de36f
paper_authors.avatarUrl: /avatars/58d44088f94cb2b5dbba099d67d8df84.svg
paper_authors.fullname: Cem Koc
paper_authors.type: user
paper_authors.user: cemkoc
paper_authors._id: 6764433e98b6fdd712b6f630
paper_authors.name: Nate True
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:57:35.922Z
paper_authors._id: 6328b013ce50f125374c1f56
paper_authors.avatarUrl: /avatars/e9d01d9e50f7cde556bf3bc37856efff.svg
paper_authors.fullname: Nate True
paper_authors.type: user
paper_authors.user: natetrue
paper_authors._id: 6764433e98b6fdd712b6f631
paper_authors.name: Albert Antony
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:57:29.611Z
paper_authors._id: 654c9ac26f2ec14d77139062
paper_authors.avatarUrl: /avatars/8a2f42591862bddf758995256a01eb38.svg
paper_authors.fullname: Albert Antony
paper_authors.type: user
paper_authors.user: albertantony
paper_authors._id: 6764433e98b6fdd712b6f632
paper_authors.name: Gokul Santhanam
paper_authors._id: 6764433e98b6fdd712b6f633
paper_authors.name: James Gabriel
paper_authors._id: 6764433e98b6fdd712b6f634
paper_authors.name: Peter Grasch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:57:03.748Z
paper_authors._id: 6649e0b6f50d471119285ab8
paper_authors.avatarUrl: /avatars/1c93d0e15f2b68281a1a2ea052450a45.svg
paper_authors.fullname: Peter Grasch
paper_authors.type: user
paper_authors.user: petergrasch
paper_authors._id: 6764433e98b6fdd712b6f635
paper_authors.name: Oncel Tuzel
paper_authors._id: 6764433e98b6fdd712b6f636
paper_authors.name: Hadi Pouransari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:56:54.004Z
paper_authors._id: 653a8e65c75dc136bfb5b0f8
paper_authors.avatarUrl: /avatars/ef252794236ce0fe2debf12773c95bb2.svg
paper_authors.fullname: Hadi Pouransari
paper_authors.type: user
paper_authors.user: hpouransari
paper_title: FastVLM: Efficient Vision Encoding for Vision Language Models
paper_summary: Scaling the input image resolution is essential for enhancing the performance
  of Vision Language Models (VLMs), particularly in text-rich image understanding
  tasks. However, popular visual encoders such as ViTs become inefficient at high
  resolutions due to the large number of tokens and high encoding latency caused
  by stacked self-attention layers. At different operational resolutions, the
  vision encoder of a VLM can be optimized along two axes: reducing encoding
  latency and minimizing the number of visual tokens passed to the LLM, thereby
  lowering overall latency. Based on a comprehensive efficiency analysis of the
  interplay between image resolution, vision latency, token count, and LLM size,
  we introduce FastVLM, a model that achieves an optimized trade-off between
  latency, model size and accuracy. FastVLM incorporates FastViTHD, a novel
  hybrid vision encoder designed to output fewer tokens and significantly reduce
  encoding time for high-resolution images. Unlike previous methods, FastVLM
  achieves the optimal balance between visual token count and image resolution
  solely by scaling the input image, eliminating the need for additional token
  pruning and simplifying the model design. In the LLaVA-1.5 setup, FastVLM
  achieves 3.2times improvement in time-to-first-token (TTFT) while
  maintaining similar performance on VLM benchmarks compared to prior works.
  Compared to LLaVa-OneVision at the highest resolution (1152times1152),
  FastVLM achieves comparable performance on key benchmarks like SeedBench and
  MMMU, using the same 0.5B LLM, but with 85times faster TTFT and a vision
  encoder that is 3.4times smaller.
paper_ai_keywords: ViTs self-attention layers FastVLM FastViTHD time-to-first-token (TTFT) SeedBench MMMU