Boosting Vision-Language Models with Transduction
<|reference_start|>Boosting Vision-Language Models with Transduction: Transduction is a powerful paradigm that leverages the structure of unlabeled data to boost predictive accuracy. We present TransCLIP, a novel and computationally efficient transductive approach designed for Vision-Language Models (VLMs). TransCLIP is applicable as a plug-and-play module on top of popular inductive zero- and few-shot models, consistently improving their performances. Our new objective function can be viewed as a regularized maximum-likelihood estimation, constrained by a KL divergence penalty that integrates the text-encoder knowledge and guides the transductive learning process. We further derive an iterative Block Majorize-Minimize (BMM) procedure for optimizing our objective, with guaranteed convergence and decoupled sample-assignment updates, yielding computationally efficient transduction for large-scale datasets. We report comprehensive evaluations, comparisons, and ablation studies that demonstrate: (i) Transduction can greatly enhance the generalization capabilities of inductive pretrained zero- and few-shot VLMs; (ii) TransCLIP substantially outperforms standard transductive few-shot learning methods relying solely on vision features, notably due to the KL-based language constraint.<|reference_end|>
@article{zanella2024boosting,
    title={Boosting Vision-Language Models with Transduction},
    author={Maxime Zanella, Beno^it G'erin, Ismail Ben Ayed},
    journal={arXiv preprint arXiv:2406.01837},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01837},
    primaryClass={cs.CV}
}