Image-Caption Encoding for Improving Zero-Shot Generalization
<|reference_start|>Image-Caption Encoding for Improving Zero-Shot Generalization: Recent advances in vision-language models have combined contrastive approaches with generative methods to achieve state-of-the-art (SOTA) on downstream inference tasks like zero-shot image classification. However, a persistent issue of these models for image classification is their out-of-distribution (OOD) generalization capabilities. We first show that when an OOD data point is misclassified, the correct class can be typically found in the Top-K predicted classes. In order to steer the model prediction toward the correct class within the top predicted classes, we propose the Image-Caption Encoding (ICE) method, a straightforward approach that directly enforces consistency between the image-conditioned and caption-conditioned predictions at evaluation time only. Intuitively, we take advantage of unique properties of the generated captions to guide our local search for the correct class label within the Top-K predicted classes. We show that our method can be easily combined with other SOTA methods to enhance Top-1 OOD accuracies by 0.5% on average and up to 3% on challenging datasets. Our code: https://github.com/Chris210634/ice<|reference_end|>
@article{yu2024image-caption,
    title={Image-Caption Encoding for Improving Zero-Shot Generalization},
    author={Eric Yang Yu, Christopher Liao, Sathvik Ravi, Theodoros Tsiligkaridis,
  Brian Kulis},
    journal={arXiv preprint arXiv:2402.02662},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.02662},
    primaryClass={cs.CV cs.CL cs.LG}
}