title: VLsI: Verbalized Layers-to-Interactions from Large to Small Vision
    Language Models
summary: The recent surge in high-quality visual instruction tuning samples from
  closed-source vision-language models (VLMs) such as GPT-4V has accelerated the
  release of open-source VLMs across various model sizes. However, scaling VLMs
  to improve performance using larger models brings significant computational
  challenges, especially for deployment on resource-constrained devices like
  mobile platforms and robots. To address this, we propose VLsI: Verbalized
  Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which
  prioritizes efficiency without compromising accuracy. VLsI leverages a unique,
  layer-wise distillation process, introducing intermediate "verbalizers" that
  map features from each layer to natural language space, allowing smaller VLMs
  to flexibly align with the reasoning processes of larger VLMs. This approach
  mitigates the training instability often encountered in output imitation and
  goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise
  progression with that of the large ones. We validate VLsI across ten
  challenging vision-language benchmarks, achieving notable performance gains
  (11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling,
  merging, or architectural changes.
paper_authors._id: 674ed11caeb5b3706525f555
paper_authors.name: Byung-Kwan Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T09:56:32.986Z
paper_authors._id: 657152eb12f162153b50ec9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
paper_authors.fullname: Byung-Kwan Lee
paper_authors.type: user
paper_authors.user: BK-Lee
paper_authors._id: 674ed11caeb5b3706525f556
paper_authors.name: Ryo Hachiuma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:21:03.967Z
paper_authors._id: 65b33e5f7cd0069ad648c4e8
paper_authors.avatarUrl: /avatars/1a746ea535cffa92ea08006e05ea414a.svg
paper_authors.fullname: Ryo Hachiuma
paper_authors.type: user
paper_authors.user: rhachiuma
paper_authors._id: 674ed11caeb5b3706525f557
paper_authors.name: Yu-Chiang Frank Wang
paper_authors._id: 674ed11caeb5b3706525f558
paper_authors.name: Yong Man Ro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:21:17.371Z
paper_authors._id: 660603529e3555d648e3baf1
paper_authors.avatarUrl: /avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
paper_authors.fullname: Yong Man Ro
paper_authors.type: user
paper_authors.user: dwightro
paper_authors._id: 674ed11caeb5b3706525f559
paper_authors.name: Yueh-Hua Wu
paper_title: VLsI: Verbalized Layers-to-Interactions from Large to Small Vision
    Language Models
paper_summary: The recent surge in high-quality visual instruction tuning samples from
  closed-source vision-language models (VLMs) such as GPT-4V has accelerated the
  release of open-source VLMs across various model sizes. However, scaling VLMs
  to improve performance using larger models brings significant computational
  challenges, especially for deployment on resource-constrained devices like
  mobile platforms and robots. To address this, we propose VLsI: Verbalized
  Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which
  prioritizes efficiency without compromising accuracy. VLsI leverages a unique,
  layer-wise distillation process, introducing intermediate "verbalizers" that
  map features from each layer to natural language space, allowing smaller VLMs
  to flexibly align with the reasoning processes of larger VLMs. This approach
  mitigates the training instability often encountered in output imitation and
  goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise
  progression with that of the large ones. We validate VLsI across ten
  challenging vision-language benchmarks, achieving notable performance gains
  (11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling,
  merging, or architectural changes.
paper_ai_keywords: vision-language models (VLMs) high-quality visual instruction tuning samples GPT-4V computational challenges resource-constrained devices mobile platforms robots VLsI: Verbalized Layers-to-Interactions layer-wise distillation verbalizers natural language space training instability output imitation vision-language benchmarks