Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning
<|reference_start|>Weak Distribution Detectors Lead to Stronger Generalizability of Vision-Language Prompt Tuning: We propose a generalized method for boosting the generalization ability of pre-trained vision-language models (VLMs) while fine-tuning on downstream few-shot tasks. The idea is realized by exploiting out-of-distribution (OOD) detection to predict whether a sample belongs to a base distribution or a novel distribution and then using the score generated by a dedicated competition based scoring function to fuse the zero-shot and few-shot classifier. The fused classifier is dynamic, which will bias towards the zero-shot classifier if a sample is more likely from the distribution pre-trained on, leading to improved base-to-novel generalization ability. Our method is performed only in test stage, which is applicable to boost existing methods without time-consuming re-training. Extensive experiments show that even weak distribution detectors can still improve VLMs' generalization ability. Specifically, with the help of OOD detectors, the harmonic mean of CoOp and ProGrad increase by 2.6 and 1.5 percentage points over 11 recognition datasets in the base-to-novel setting.<|reference_end|>
@article{ding2024weak,
    title={Weak Distribution Detectors Lead to Stronger Generalizability of
  Vision-Language Prompt Tuning},
    author={Kun Ding and Haojian Zhang and Qiang Yu and Ying Wang and Shiming
  Xiang and Chunhong Pan},
    journal={arXiv preprint arXiv:2404.00603},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.00603},
    primaryClass={cs.CV}
}