title: Symmetrical Visual Contrastive Optimization: Aligning Vision-Language
    Models with Minimal Contrastive Images
summary: Recent studies have shown that Large Vision-Language Models (VLMs) tend to
  neglect image content and over-rely on language-model priors, resulting in
  errors in visually grounded tasks and hallucinations. We hypothesize that this
  issue arises because existing VLMs are not explicitly trained to generate texts
  that are accurately grounded in fine-grained image details. To enhance visual
  feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive
  Optimization), a novel finetuning objective that steers the model toward
  capturing important visual details and aligning them with corresponding text
  tokens. To further facilitate this detailed alignment, we introduce MVC, a
  paired image-text dataset built by automatically filtering and augmenting
  visual counterfactual data to challenge the model with hard contrastive cases
  involving Minimal Visual Contrasts. Experiments show that our method
  consistently improves VLM performance across diverse benchmarks covering
  various abilities and domains, achieving up to a 22% reduction in
  hallucinations, and significant gains in vision-centric and general tasks.
  Notably, these improvements become increasingly pronounced in benchmarks with
  higher visual dependency. In short, S-VCO offers a significant enhancement of
  VLM's visually-dependent task performance while retaining or even improving the
  model's general abilities. We opensource our code at https://s-vco.github.io/
paper_authors._id: 67b7cdac904136d47c3966d8
paper_authors.name: Shengguang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:59:21.001Z
paper_authors._id: 65222f97ef06bb99753cb829
paper_authors.avatarUrl: /avatars/f1a743d74e6d38b916acaec91b4e7e4f.svg
paper_authors.fullname: Shengguang Wu
paper_authors.type: user
paper_authors.user: danielwusg
paper_authors._id: 67b7cdac904136d47c3966d9
paper_authors.name: Fan-Yun Sun
paper_authors._id: 67b7cdac904136d47c3966da
paper_authors.name: Kaiyue Wen
paper_authors._id: 67b7cdac904136d47c3966db
paper_authors.name: Nick Haber
paper_title: Symmetrical Visual Contrastive Optimization: Aligning Vision-Language
    Models with Minimal Contrastive Images
paper_summary: Recent studies have shown that Large Vision-Language Models (VLMs) tend to
  neglect image content and over-rely on language-model priors, resulting in
  errors in visually grounded tasks and hallucinations. We hypothesize that this
  issue arises because existing VLMs are not explicitly trained to generate texts
  that are accurately grounded in fine-grained image details. To enhance visual
  feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive
  Optimization), a novel finetuning objective that steers the model toward
  capturing important visual details and aligning them with corresponding text
  tokens. To further facilitate this detailed alignment, we introduce MVC, a
  paired image-text dataset built by automatically filtering and augmenting
  visual counterfactual data to challenge the model with hard contrastive cases
  involving Minimal Visual Contrasts. Experiments show that our method
  consistently improves VLM performance across diverse benchmarks covering
  various abilities and domains, achieving up to a 22% reduction in
  hallucinations, and significant gains in vision-centric and general tasks.
  Notably, these improvements become increasingly pronounced in benchmarks with
  higher visual dependency. In short, S-VCO offers a significant enhancement of
  VLM's visually-dependent task performance while retaining or even improving the
  model's general abilities. We opensource our code at https://s-vco.github.io/
paper_ai_keywords: Large Vision-Language Models (VLMs) hallucinations fine-grained image details text tokens Symmetrical Visual Contrastive Optimization (S-VCO) paired image-text dataset visual counterfactual data Minimal Visual Contrasts vision-centric tasks