Few-shot Adaptation of Medical Vision-Language Models
<|reference_start|>Few-shot Adaptation of Medical Vision-Language Models: Integrating image and text data through multi-modal learning has emerged as a new approach in medical imaging research, following its successful deployment in computer vision. While considerable efforts have been dedicated to establishing medical foundation models and their zero-shot transfer to downstream tasks, the popular few-shot setting remains relatively unexplored. Following on from the currently strong emergence of this setting in computer vision, we introduce the first structured benchmark for adapting medical vision-language models (VLMs) in a strict few-shot regime and investigate various adaptation strategies commonly used in the context of natural images. Furthermore, we evaluate a simple generalization of the linear-probe adaptation baseline, which seeks an optimal blending of the visual prototypes and text embeddings via learnable class-wise multipliers. Surprisingly, such a text-informed linear probe yields competitive performances in comparison to convoluted prompt-learning and adapter-based strategies, while running considerably faster and accommodating the black-box setting. Our extensive experiments span three different medical modalities and specialized foundation models, nine downstream tasks, and several state-of-the-art few-shot adaptation methods. We made our benchmark and code publicly available to trigger further developments in this emergent subject: \url{https://github.com/FereshteShakeri/few-shot-MedVLMs}.<|reference_end|>
@article{shakeri2024few-shot,
    title={Few-shot Adaptation of Medical Vision-Language Models},
    author={Fereshteh Shakeri, Yunshi Huang, Julio Silva-Rodr'iguez, Houda Bahig,
  An Tang, Jose Dolz, Ismail Ben Ayed},
    journal={arXiv preprint arXiv:2409.03868},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.03868},
    primaryClass={cs.CV}
}