VLsI: Verbalized Layers-to-Interactions from Large to Small Vision
  Language Models
The recent surge in high-quality visual instruction tuning samples from
closed-source vision-language models (VLMs) such as GPT-4V has accelerated the
release of open-source VLMs across various model sizes. However, scaling VLMs
to improve performance using larger models brings significant computational
challenges, especially for deployment on resource-constrained devices like
mobile platforms and robots. To address this, we propose VLsI: Verbalized
Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which
prioritizes efficiency without compromising accuracy. VLsI leverages a unique,
layer-wise distillation process, introducing intermediate "verbalizers" that
map features from each layer to natural language space, allowing smaller VLMs
to flexibly align with the reasoning processes of larger VLMs. This approach
mitigates the training instability often encountered in output imitation and
goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise
progression with that of the large ones. We validate VLsI across ten
challenging vision-language benchmarks, achieving notable performance gains
(11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling,
merging, or architectural changes.
674ed11caeb5b3706525f555
Byung-Kwan Lee
claimed_verified
2024-12-03T09:56:32.986Z
657152eb12f162153b50ec9d
https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
Byung-Kwan Lee
user
BK-Lee
674ed11caeb5b3706525f556
Ryo Hachiuma
admin_assigned
2024-12-03T15:21:03.967Z
65b33e5f7cd0069ad648c4e8
/avatars/1a746ea535cffa92ea08006e05ea414a.svg
Ryo Hachiuma
user
rhachiuma
674ed11caeb5b3706525f557
Yu-Chiang Frank Wang
674ed11caeb5b3706525f558
Yong Man Ro
admin_assigned
2024-12-03T15:21:17.371Z
660603529e3555d648e3baf1
/avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
Yong Man Ro
user
dwightro
674ed11caeb5b3706525f559
Yueh-Hua Wu
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision
  Language Models
The recent surge in high-quality visual instruction tuning samples from
closed-source vision-language models (VLMs) such as GPT-4V has accelerated the
release of open-source VLMs across various model sizes. However, scaling VLMs
to improve performance using larger models brings significant computational
challenges, especially for deployment on resource-constrained devices like
mobile platforms and robots. To address this, we propose VLsI: Verbalized
Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which
prioritizes efficiency without compromising accuracy. VLsI leverages a unique,
layer-wise distillation process, introducing intermediate "verbalizers" that
map features from each layer to natural language space, allowing smaller VLMs
to flexibly align with the reasoning processes of larger VLMs. This approach
mitigates the training instability often encountered in output imitation and
goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise
progression with that of the large ones. We validate VLsI across ten
challenging vision-language benchmarks, achieving notable performance gains
(11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling,
merging, or architectural changes.
VLsI, a new family of VLMs, uses layer-wise distillation with verbalizers to enhance performance on resource-constrained devices without scaling model size.
vision-language models VLMs GPT-4V VLsI Verbalized Layers-to-Interactions layer-wise distillation verbalizers natural language space training instability final-layer tuning vision-language benchmarks