title: Discriminative Fine-tuning of LVLMs
summary: Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the
  de facto approach for discriminative vision-language representation learning.
  However, these models have limited language understanding, often exhibiting a
  "bag of words" behavior. At the same time, Large Vision-Language Models
  (LVLMs), which combine vision encoders with LLMs, have been shown capable of
  detailed vision-language reasoning, yet their autoregressive nature renders
  them less suitable for discriminative tasks.
    In this work, we propose to combine "the best of both worlds": a new training
  approach for discriminative fine-tuning of LVLMs that results in strong
  discriminative and compositional capabilities. Essentially, our approach
  converts a generative LVLM into a discriminative one, unlocking its capability
  for powerful image-text discrimination combined with enhanced language
  understanding.
    Our contributions include: (1) A carefully designed training/optimization
  framework that utilizes image-text pairs of variable length and granularity for
  training the model with both contrastive and next-token prediction losses. This
  is accompanied by ablation studies that justify the necessity of our
  framework's components. (2) A parameter-efficient adaptation method using a
  combination of soft prompting and LoRA adapters. (3) Significant improvements
  over state-of-the-art CLIP-like models of similar size, including standard
  image-text retrieval benchmarks and notable gains in compositionality.
paper_authors._id: 6752e52e8b6b5be84ef0065b
paper_authors.name: Yassine Ouali
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:55:38.196Z
paper_authors._id: 60d5fc283039da71be392aaf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624636332039-noauth.png
paper_authors.fullname: Yassine Ouali
paper_authors.type: user
paper_authors.user: youali
paper_authors._id: 6752e52e8b6b5be84ef0065c
paper_authors.name: Adrian Bulat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:55:40.423Z
paper_authors._id: 64a6e0923987f4dd3c37087d
paper_authors.avatarUrl: /avatars/88bc86ddce1b38a012b07c81f5c61183.svg
paper_authors.fullname: Adrian Bulat
paper_authors.type: user
paper_authors.user: adrianb1
paper_authors._id: 6752e52e8b6b5be84ef0065d
paper_authors.name: Alexandros Xenos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T15:55:36.081Z
paper_authors._id: 62430990422d7ec108c0c57d
paper_authors.avatarUrl: /avatars/23ed460f9c544252d904a670c3a8fff1.svg
paper_authors.fullname: Alexandros Xenos
paper_authors.type: user
paper_authors.user: alxenos
paper_authors._id: 6752e52e8b6b5be84ef0065e
paper_authors.name: Anestis Zaganidis
paper_authors._id: 6752e52e8b6b5be84ef0065f
paper_authors.name: Ioannis Maniadis Metaxas
paper_authors._id: 6752e52e8b6b5be84ef00660
paper_authors.name: Georgios Tzimiropoulos
paper_authors._id: 6752e52e8b6b5be84ef00661
paper_authors.name: Brais Martinez
paper_title: Discriminative Fine-tuning of LVLMs
paper_summary: Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the
  de facto approach for discriminative vision-language representation learning.
  However, these models have limited language understanding, often exhibiting a
  "bag of words" behavior. At the same time, Large Vision-Language Models
  (LVLMs), which combine vision encoders with LLMs, have been shown capable of
  detailed vision-language reasoning, yet their autoregressive nature renders
  them less suitable for discriminative tasks.
    In this work, we propose to combine "the best of both worlds": a new training
  approach for discriminative fine-tuning of LVLMs that results in strong
  discriminative and compositional capabilities. Essentially, our approach
  converts a generative LVLM into a discriminative one, unlocking its capability
  for powerful image-text discrimination combined with enhanced language
  understanding.
    Our contributions include: (1) A carefully designed training/optimization
  framework that utilizes image-text pairs of variable length and granularity for
  training the model with both contrastive and next-token prediction losses. This
  is accompanied by ablation studies that justify the necessity of our
  framework's components. (2) A parameter-efficient adaptation method using a
  combination of soft prompting and LoRA adapters. (3) Significant improvements
  over state-of-the-art CLIP-like models of similar size, including standard
  image-text retrieval benchmarks and notable gains in compositionality.
paper_ai_summary: A novel training approach converts Large Vision-Language Models into discriminative ones, enhancing image-text discrimination and language understanding.
paper_ai_keywords: Contrastively-trained Vision-Language Models CLIP Large Vision-Language Models LVLMs discriminative fine-tuning generative discriminative image-text pairs contrastive losses next-token prediction losses ablation studies parameter-efficient adaptation soft prompting LoRA adapters image-text retrieval benchmarks compositionality