The Hard Positive Truth about Vision-Language Compositionality
<|reference_start|>The Hard Positive Truth about Vision-Language Compositionality: Several benchmarks have concluded that our best vision-language models (e.g., CLIP) are lacking in compositionality. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors. In response, a surge of recent proposals show improvements by finetuning CLIP with distractors as hard negatives. Our investigations reveal that these improvements have, in fact, been significantly overstated -- because existing benchmarks do not probe whether finetuned vision-language models remain invariant to hard positives. By curating an evaluation dataset with 112,382 hard negatives and hard positives, we uncover that including hard positives decreases CLIP's performance by 12.9%, while humans perform effortlessly at 99%. CLIP finetuned with hard negatives results in an even larger decrease, up to 38.7%. With this finding, we then produce a 1,775,259 image-text training set with both hard negative and hard positive captions. By training with both, we see improvements on existing benchmarks while simultaneously improving performance on hard positives, indicating a more robust improvement in compositionality. Our work suggests the need for future research to rigorously test and improve CLIP's understanding of semantic relationships between related "positive" concepts.<|reference_end|>
@article{kamath2024the,
    title={The Hard Positive Truth about Vision-Language Compositionality},
    author={Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, Ranjay Krishna},
    journal={arXiv preprint arXiv:2409.17958},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.17958},
    primaryClass={cs.CL cs.CV}
}