title: OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary
    Embedding Distillation
summary: The standard practice for developing contemporary MLLMs is to feed features
  from vision encoder(s) into the LLM and train with natural language
  supervision. In this work, we posit an overlooked opportunity to optimize the
  intermediate LLM representations through a vision perspective (objective),
  i.e., solely natural language supervision is sub-optimal for the MLLM's visual
  understanding ability. To that end, we propose OLA-VLM, the first approach
  distilling knowledge into the LLM's hidden representations from a set of target
  visual representations. Firstly, we formulate the objective during the
  pretraining stage in MLLMs as a coupled optimization of predictive visual
  embedding and next text-token prediction. Secondly, we investigate MLLMs
  trained solely with natural language supervision and identify a positive
  correlation between the quality of visual representations within these models
  and their downstream performance. Moreover, upon probing our OLA-VLM, we
  observe improved representation quality owing to the embedding optimization.
  Thirdly, we demonstrate that our OLA-VLM outperforms the single and
  multi-encoder baselines, proving our approach's superiority over explicitly
  feeding the corresponding features to the LLM. Particularly, OLA-VLM boosts
  performance by an average margin of up to 2.5% on various benchmarks, with a
  notable improvement of 8.7% on the Depth task in CV-Bench. Our code is
  open-sourced at https://github.com/SHI-Labs/OLA-VLM .
paper_authors._id: 675ba1257950eafb111baff3
paper_authors.name: Jitesh Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:22:31.309Z
paper_authors._id: 623dfe96dcda6a715304cbca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623dfe96dcda6a715304cbca/B7V_IbQNXAcotOKE_mJQ4.jpeg
paper_authors.fullname: Jitesh Jain
paper_authors.type: user
paper_authors.user: praeclarumjj3
paper_authors._id: 675ba1257950eafb111baff4
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:21:16.481Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 675ba1257950eafb111baff5
paper_authors.name: Humphrey Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:21:22.020Z
paper_authors._id: 61e1188afc27c0f5e3641eb3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e1188afc27c0f5e3641eb3/R4JFlWgYomZFvZ9CLSZfD.jpeg
paper_authors.fullname: Humphrey Shi
paper_authors.type: user
paper_authors.user: Humphrey
paper_authors._id: 675ba1257950eafb111baff6
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:21:27.791Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 675ba1257950eafb111baff7
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T11:35:08.547Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_title: OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary
    Embedding Distillation
paper_summary: The standard practice for developing contemporary MLLMs is to feed features
  from vision encoder(s) into the LLM and train with natural language
  supervision. In this work, we posit an overlooked opportunity to optimize the
  intermediate LLM representations through a vision perspective (objective),
  i.e., solely natural language supervision is sub-optimal for the MLLM's visual
  understanding ability. To that end, we propose OLA-VLM, the first approach
  distilling knowledge into the LLM's hidden representations from a set of target
  visual representations. Firstly, we formulate the objective during the
  pretraining stage in MLLMs as a coupled optimization of predictive visual
  embedding and next text-token prediction. Secondly, we investigate MLLMs
  trained solely with natural language supervision and identify a positive
  correlation between the quality of visual representations within these models
  and their downstream performance. Moreover, upon probing our OLA-VLM, we
  observe improved representation quality owing to the embedding optimization.
  Thirdly, we demonstrate that our OLA-VLM outperforms the single and
  multi-encoder baselines, proving our approach's superiority over explicitly
  feeding the corresponding features to the LLM. Particularly, OLA-VLM boosts
  performance by an average margin of up to 2.5% on various benchmarks, with a
  notable improvement of 8.7% on the Depth task in CV-Bench. Our code is
  open-sourced at https://github.com/SHI-Labs/OLA-VLM .
paper_ai_keywords: MLLMs (Multimodal Large Language Models) VLM (Vision-Language Model) OLA-VLM (Optimized Latent Alignment Vision-Language Model) predictive visual embedding next text-token prediction representation quality downstream performance single encoder multi-encoder CV-Bench (Computer Vision Benchmark)