CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical Imaging
<|reference_start|>CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical Imaging: Existing vision-text contrastive learning models enhance representation transferability and support zero-shot prediction by matching paired image and caption embeddings while pushing unrelated pairs apart. However, astronomical image-label datasets are significantly smaller compared to general image and label datasets available from the internet. We introduce CosmoCLIP, an astronomical image-text contrastive learning framework precisely fine-tuned on the pre-trained CLIP model using SpaceNet and BLIP-based captions. SpaceNet, attained via FLARE, constitutes ~13k optimally distributed images, while BLIP acts as a rich knowledge extractor. The rich semantics derived from this SpaceNet and BLIP descriptions, when learned contrastively, enable CosmoCLIP to achieve superior generalization across various in-domain and out-of-domain tasks. Our results demonstrate that CosmoCLIP is a straightforward yet powerful framework, significantly outperforming CLIP in zero-shot classification and image-text retrieval tasks.<|reference_end|>
@article{imam2024cosmoclip:,
    title={CosmoCLIP: Generalizing Large Vision-Language Models for Astronomical
  Imaging},
    author={Raza Imam, Mohammed Talha Alam, Umaima Rahman, Mohsen Guizani, Fakhri
  Karray},
    journal={arXiv preprint arXiv:2407.07315},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.07315},
    primaryClass={cs.CV}
}