MedUnA: Language guided Unsupervised Adaptation of Vision-Language Models for Medical Image Classification
<|reference_start|>MedUnA: Language guided Unsupervised Adaptation of Vision-Language Models for Medical Image Classification: In medical image classification, supervised learning is challenging due to the lack of labeled medical images. Contrary to the traditional \textit{modus operandi} of pre-training followed by fine-tuning, this work leverages the visual-textual alignment within Vision-Language models (\texttt{VLMs}) to facilitate the unsupervised learning.   Specifically, we propose \underline{Med}ical \underline{Un}supervised \underline{A}daptation (\texttt{MedUnA}), constituting two-stage training: Adapter Pre-training, and Unsupervised Learning. In the first stage, we use descriptions generated by a Large Language Model (\texttt{LLM}) corresponding to class labels, which are passed through the text encoder \texttt{BioBERT}. The resulting text embeddings are then aligned with the class labels by training a lightweight \texttt{adapter}. We choose \texttt{\texttt{LLMs}} because of their capability to generate detailed, contextually relevant descriptions to obtain enhanced text embeddings.   In the second stage, the trained \texttt{adapter} is integrated with the visual encoder of \texttt{MedCLIP}. This stage employs a contrastive entropy-based loss and prompt tuning to align visual embeddings. We incorporate self-entropy minimization into the overall training objective to ensure more confident embeddings, which are crucial for effective unsupervised learning and alignment. We evaluate the performance of \texttt{MedUnA} on three different kinds of data modalities - chest X-rays, eye fundus and skin lesion images. The results demonstrate significant accuracy gain on average compared to the baselines across different datasets, highlighting the efficacy of our approach.<|reference_end|>
@article{rahman2024meduna:,
    title={MedUnA: Language guided Unsupervised Adaptation of Vision-Language
  Models for Medical Image Classification},
    author={Umaima Rahman, Raza Imam, Dwarikanath Mahapatra, Boulbaba Ben Amor},
    journal={arXiv preprint arXiv:2409.02729},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.02729},
    primaryClass={cs.CV}
}