MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment
<|reference_start|>MeDSLIP: Medical Dual-Stream Language-Image Pre-training for Fine-grained Alignment: Vision-language pre-training (VLP) models have shown significant advancements in the medical domain. Yet, most VLP models align raw reports to images at a very coarse level, without modeling fine-grained relationships between anatomical and pathological concepts outlined in reports and the corresponding semantic counterparts in images. To address this problem, we propose a Medical Dual-Stream Language-Image Pre-training (MeDSLIP) framework. Specifically, MeDSLIP establishes vision-language fine-grained alignments via disentangling visual and textual representations into anatomy-relevant and pathology-relevant streams. Moreover, a novel vision-language Prototypical Contr-astive Learning (ProtoCL) method is adopted in MeDSLIP to enhance the alignment within the anatomical and pathological streams. MeDSLIP further employs cross-stream Intra-image Contrastive Learning (ICL) to ensure the consistent coexistence of paired anatomical and pathological concepts within the same image. Such a cross-stream regularization encourages the model to exploit the synchrony between two streams for a more comprehensive representation learning. MeDSLIP is evaluated under zero-shot and supervised fine-tuning settings on three public datasets: NIH CXR14, RSNA Pneumonia, and SIIM-ACR Pneumothorax. Under these settings, MeDSLIP outperforms six leading CNN-based models on classification, grounding, and segmentation tasks.<|reference_end|>
@article{fan2024medslip:,
    title={MeDSLIP: Medical Dual-Stream Language-Image Pre-training for
  Fine-grained Alignment},
    author={Wenrui Fan, Mohammod Naimul Islam Suvon, Shuo Zhou, Xianyuan Liu,
  Samer Alabed, Venet Osmani, Andrew Swift, Chen Chen, Haiping Lu},
    journal={arXiv preprint arXiv:2403.10635},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.10635},
    primaryClass={cs.CV}
}