Multimodal CLIP Inference for Meta-Few-Shot Image Classification
<|reference_start|>Multimodal CLIP Inference for Meta-Few-Shot Image Classification: In recent literature, few-shot classification has predominantly been defined by the N-way k-shot meta-learning problem. Models designed for this purpose are usually trained to excel on standard benchmarks following a restricted setup, excluding the use of external data. Given the recent advancements in large language and vision models, a question naturally arises: can these models directly perform well on meta-few-shot learning benchmarks? Multimodal foundation models like CLIP, which learn a joint (image, text) embedding, are of particular interest. Indeed, multimodal training has proven to enhance model robustness, especially regarding ambiguities, a limitation frequently observed in the few-shot setup. This study demonstrates that combining modalities from CLIP's text and image encoders outperforms state-of-the-art meta-few-shot learners on widely adopted benchmarks, all without additional training. Our results confirm the potential and robustness of multimodal foundation models like CLIP and serve as a baseline for existing and future approaches leveraging such models.<|reference_end|>
@article{ferragu2024multimodal,
    title={Multimodal CLIP Inference for Meta-Few-Shot Image Classification},
    author={Constance Ferragu, Philomene Chagniot, Vincent Coyette},
    journal={arXiv preprint arXiv:2405.10954},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.10954},
    primaryClass={cs.CV}
}