OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary
  Embedding Distillation
The standard practice for developing contemporary MLLMs is to feed features
from vision encoder(s) into the LLM and train with natural language
supervision. In this work, we posit an overlooked opportunity to optimize the
intermediate LLM representations through a vision perspective (objective),
i.e., solely natural language supervision is sub-optimal for the MLLM's visual
understanding ability. To that end, we propose OLA-VLM, the first approach
distilling knowledge into the LLM's hidden representations from a set of target
visual representations. Firstly, we formulate the objective during the
pretraining stage in MLLMs as a coupled optimization of predictive visual
embedding and next text-token prediction. Secondly, we investigate MLLMs
trained solely with natural language supervision and identify a positive
correlation between the quality of visual representations within these models
and their downstream performance. Moreover, upon probing our OLA-VLM, we
observe improved representation quality owing to the embedding optimization.
Thirdly, we demonstrate that our OLA-VLM outperforms the single and
multi-encoder baselines, proving our approach's superiority over explicitly
feeding the corresponding features to the LLM. Particularly, OLA-VLM boosts
performance by an average margin of up to 2.5% on various benchmarks, with a
notable improvement of 8.7% on the Depth task in CV-Bench. Our code is
open-sourced at https://github.com/SHI-Labs/OLA-VLM .
675ba1257950eafb111baff3
Jitesh Jain
claimed_verified
2024-12-13T15:22:31.309Z
623dfe96dcda6a715304cbca
https://cdn-avatars.huggingface.co/v1/production/uploads/623dfe96dcda6a715304cbca/B7V_IbQNXAcotOKE_mJQ4.jpeg
Jitesh Jain
user
praeclarumjj3
675ba1257950eafb111baff4
Zhengyuan Yang
admin_assigned
2024-12-16T11:21:16.481Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
675ba1257950eafb111baff5
Humphrey Shi
admin_assigned
2024-12-16T11:21:22.020Z
61e1188afc27c0f5e3641eb3
https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
Humphrey Shi
user
Humphrey
675ba1257950eafb111baff6
Jianfeng Gao
admin_assigned
2024-12-16T11:21:27.791Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
675ba1257950eafb111baff7
Jianwei Yang
admin_assigned
2024-12-16T11:35:08.547Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary
  Embedding Distillation
The standard practice for developing contemporary MLLMs is to feed features
from vision encoder(s) into the LLM and train with natural language
supervision. In this work, we posit an overlooked opportunity to optimize the
intermediate LLM representations through a vision perspective (objective),
i.e., solely natural language supervision is sub-optimal for the MLLM's visual
understanding ability. To that end, we propose OLA-VLM, the first approach
distilling knowledge into the LLM's hidden representations from a set of target
visual representations. Firstly, we formulate the objective during the
pretraining stage in MLLMs as a coupled optimization of predictive visual
embedding and next text-token prediction. Secondly, we investigate MLLMs
trained solely with natural language supervision and identify a positive
correlation between the quality of visual representations within these models
and their downstream performance. Moreover, upon probing our OLA-VLM, we
observe improved representation quality owing to the embedding optimization.
Thirdly, we demonstrate that our OLA-VLM outperforms the single and
multi-encoder baselines, proving our approach's superiority over explicitly
feeding the corresponding features to the LLM. Particularly, OLA-VLM boosts
performance by an average margin of up to 2.5% on various benchmarks, with a
notable improvement of 8.7% on the Depth task in CV-Bench. Our code is
open-sourced at https://github.com/SHI-Labs/OLA-VLM .
OLA-VLM optimizes intermediate MLLM representations using visual knowledge distillation, improving visual understanding and downstream performance better than traditional methods.
MLLMs vision encoder natural language supervision OLA-VLM pretraining stage coupled optimization predictive visual embedding next text-token prediction visual representations downstream performance representation quality single and multi-encoder baselines CV-Bench Depth task