Jina CLIP: Your CLIP Model Is Also Your Text Retriever
<|reference_start|>Jina CLIP: Your CLIP Model Is Also Your Text Retriever: Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-clip-v1 model to achieve the state-of-the-art performance on both text-image and text-text retrieval tasks.<|reference_end|>
@article{koukounas2024jina,
    title={Jina CLIP: Your CLIP Model Is Also Your Text Retriever},
    author={Andreas Koukounas, Georgios Mastrapas, Michael G"unther, Bo Wang,
  Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan
  Fontanals Mart'inez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan
  Wang, Han Xiao},
    journal={arXiv preprint arXiv:2405.20204},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.20204},
    primaryClass={cs.CL cs.AI cs.CV cs.IR}
}