CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs
<|reference_start|>CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing Hallucinations in LVLMs: Despite recent successes, LVLMs or Large Vision Language Models are prone to hallucinating details like objects and their properties or relations, limiting their real-world deployment. To address this and improve their robustness, we present CLIP-DPO, a preference optimization method that leverages contrastively pre-trained Vision-Language (VL) embedding models, such as CLIP, for DPO-based optimization of LVLMs. Unlike prior works tackling LVLM hallucinations, our method does not rely on paid-for APIs, and does not require additional training data or the deployment of other external LVLMs. Instead, starting from the initial pool of supervised fine-tuning data, we generate a diverse set of predictions, which are ranked based on their CLIP image-text similarities, and then filtered using a robust rule-based approach to obtain a set of positive and negative pairs for DPO-based training. We applied CLIP-DPO fine-tuning to the MobileVLM-v2 family of models and to LlaVA-1.5, in all cases observing significant improvements in terms of hallucination reduction over baseline models. We also observe better performance for zero-shot classification, suggesting improved grounding capabilities, and verify that the original performance on standard LVLM benchmarks is overall preserved.<|reference_end|>
@article{ouali2024clip-dpo:,
    title={CLIP-DPO: Vision-Language Models as a Source of Preference for Fixing
  Hallucinations in LVLMs},
    author={Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios
  Tzimiropoulos},
    journal={arXiv preprint arXiv:2408.10433},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.10433},
    primaryClass={cs.CV}
}