On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?
<|reference_start|>On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?: The development of large vision-language models, notably CLIP, has catalyzed research into effective adaptation techniques, with a particular focus on soft prompt tuning. Conjointly, test-time augmentation, which utilizes multiple augmented views of a single image to enhance zero-shot generalization, is emerging as a significant area of interest. This has predominantly directed research efforts toward test-time prompt tuning. In contrast, we introduce a robust MeanShift for Test-time Augmentation (MTA), which surpasses prompt-based methods without requiring this intensive training procedure. This positions MTA as an ideal solution for both standalone and API-based applications. Additionally, our method does not rely on ad hoc rules (e.g., confidence threshold) used in some previous test-time augmentation techniques to filter the augmented views. Instead, MTA incorporates a quality assessment variable for each view directly into its optimization process, termed as the inlierness score. This score is jointly optimized with a density mode seeking process, leading to an efficient training- and hyperparameter-free approach. We extensively benchmark our method on 15 datasets and demonstrate MTA's superiority and computational efficiency. Deployed easily as plug-and-play module on top of zero-shot models and state-of-the-art few-shot methods, MTA shows systematic and consistent improvements.<|reference_end|>
@article{zanella2024on,
    title={On the test-time zero-shot generalization of vision-language models: Do
  we really need prompt learning?},
    author={Maxime Zanella, Ismail Ben Ayed},
    journal={arXiv preprint arXiv:2405.02266},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.02266},
    primaryClass={cs.CV}
}