Discriminative Fine-tuning of LVLMs
Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the
de facto approach for discriminative vision-language representation learning.
However, these models have limited language understanding, often exhibiting a
"bag of words" behavior. At the same time, Large Vision-Language Models
(LVLMs), which combine vision encoders with LLMs, have been shown capable of
detailed vision-language reasoning, yet their autoregressive nature renders
them less suitable for discriminative tasks.
  In this work, we propose to combine "the best of both worlds": a new training
approach for discriminative fine-tuning of LVLMs that results in strong
discriminative and compositional capabilities. Essentially, our approach
converts a generative LVLM into a discriminative one, unlocking its capability
for powerful image-text discrimination combined with enhanced language
understanding.
  Our contributions include: (1) A carefully designed training/optimization
framework that utilizes image-text pairs of variable length and granularity for
training the model with both contrastive and next-token prediction losses. This
is accompanied by ablation studies that justify the necessity of our
framework's components. (2) A parameter-efficient adaptation method using a
combination of soft prompting and LoRA adapters. (3) Significant improvements
over state-of-the-art CLIP-like models of similar size, including standard
image-text retrieval benchmarks and notable gains in compositionality.
6752e52e8b6b5be84ef0065b
Yassine Ouali
claimed_verified
2024-12-06T15:55:38.196Z
60d5fc283039da71be392aaf
https://cdn-avatars.huggingface.co/v1/production/uploads/1624636332039-noauth.png
Yassine Ouali
user
youali
6752e52e8b6b5be84ef0065c
Adrian Bulat
claimed_verified
2024-12-06T15:55:40.423Z
64a6e0923987f4dd3c37087d
/avatars/88bc86ddce1b38a012b07c81f5c61183.svg
Adrian Bulat
user
adrianb1
6752e52e8b6b5be84ef0065d
Alexandros Xenos
claimed_verified
2024-12-06T15:55:36.081Z
62430990422d7ec108c0c57d
/avatars/23ed460f9c544252d904a670c3a8fff1.svg
Alexandros Xenos
user
alxenos
6752e52e8b6b5be84ef0065e
Anestis Zaganidis
6752e52e8b6b5be84ef0065f
Ioannis Maniadis Metaxas
6752e52e8b6b5be84ef00660
Georgios Tzimiropoulos
6752e52e8b6b5be84ef00661
Brais Martinez
Discriminative Fine-tuning of LVLMs
Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the
de facto approach for discriminative vision-language representation learning.
However, these models have limited language understanding, often exhibiting a
"bag of words" behavior. At the same time, Large Vision-Language Models
(LVLMs), which combine vision encoders with LLMs, have been shown capable of
detailed vision-language reasoning, yet their autoregressive nature renders
them less suitable for discriminative tasks.
  In this work, we propose to combine "the best of both worlds": a new training
approach for discriminative fine-tuning of LVLMs that results in strong
discriminative and compositional capabilities. Essentially, our approach
converts a generative LVLM into a discriminative one, unlocking its capability
for powerful image-text discrimination combined with enhanced language
understanding.
  Our contributions include: (1) A carefully designed training/optimization
framework that utilizes image-text pairs of variable length and granularity for
training the model with both contrastive and next-token prediction losses. This
is accompanied by ablation studies that justify the necessity of our
framework's components. (2) A parameter-efficient adaptation method using a
combination of soft prompting and LoRA adapters. (3) Significant improvements
over state-of-the-art CLIP-like models of similar size, including standard
image-text retrieval benchmarks and notable gains in compositionality.
A novel training approach converts Large Vision-Language Models into discriminative ones, enhancing image-text discrimination and language understanding.
Contrastively-trained Vision-Language Models CLIP Large Vision-Language Models LVLMs discriminative fine-tuning generative discriminative image-text pairs contrastive losses next-token prediction losses ablation studies parameter-efficient adaptation soft prompting LoRA adapters image-text retrieval benchmarks compositionality