title: OViP: Online Vision-Language Preference Learning
summary: Large vision-language models (LVLMs) remain vulnerable to hallucination,
  often generating content misaligned with visual inputs. While recent approaches
  advance multi-modal Direct Preference Optimization (DPO) to mitigate
  hallucination, they typically rely on predefined or randomly edited negative
  samples that fail to reflect actual model errors, limiting training efficacy.
  In this work, we propose an Online Vision-language Preference Learning (OViP)
  framework that dynamically constructs contrastive training data based on the
  model's own hallucinated outputs. By identifying semantic differences between
  sampled response pairs and synthesizing negative images using a diffusion
  model, OViP generates more relevant supervision signals in real time. This
  failure-driven training enables adaptive alignment of both textual and visual
  preferences. Moreover, we refine existing evaluation protocols to better
  capture the trade-off between hallucination suppression and expressiveness.
  Experiments on hallucination and general benchmarks demonstrate that OViP
  effectively reduces hallucinations while preserving core multi-modal
  capabilities.
paper_authors._id: 682fdcc0087ea62f1663df96
paper_authors.name: Shujun Liu
paper_authors._id: 682fdcc0087ea62f1663df97
paper_authors.name: Siyuan Wang
paper_authors._id: 682fdcc0087ea62f1663df98
paper_authors.name: Zejun Li
paper_authors._id: 682fdcc0087ea62f1663df99
paper_authors.name: Jianxiang Wang
paper_authors._id: 682fdcc0087ea62f1663df9a
paper_authors.name: Cheng Zeng
paper_authors._id: 682fdcc0087ea62f1663df9b
paper_authors.name: Zhongyu Wei
paper_title: OViP: Online Vision-Language Preference Learning
paper_summary: Large vision-language models (LVLMs) remain vulnerable to hallucination,
  often generating content misaligned with visual inputs. While recent approaches
  advance multi-modal Direct Preference Optimization (DPO) to mitigate
  hallucination, they typically rely on predefined or randomly edited negative
  samples that fail to reflect actual model errors, limiting training efficacy.
  In this work, we propose an Online Vision-language Preference Learning (OViP)
  framework that dynamically constructs contrastive training data based on the
  model's own hallucinated outputs. By identifying semantic differences between
  sampled response pairs and synthesizing negative images using a diffusion
  model, OViP generates more relevant supervision signals in real time. This
  failure-driven training enables adaptive alignment of both textual and visual
  preferences. Moreover, we refine existing evaluation protocols to better
  capture the trade-off between hallucination suppression and expressiveness.
  Experiments on hallucination and general benchmarks demonstrate that OViP
  effectively reduces hallucinations while preserving core multi-modal
  capabilities.
paper_ai_summary: OViP dynamically generates contrastive training data using a diffusion model to reduce hallucinations in large vision-language models while maintaining their multi-modal capabilities.
paper_ai_keywords: large vision-language models hallucination multi-modal Direct Preference Optimization OViP diffusion model contrastive training semantic differences failure-driven training