title: Jina CLIP: Your CLIP Model Is Also Your Text Retriever
summary: Contrastive Language-Image Pretraining (CLIP) is widely used to train models
  to align images and texts in a common embedding space by mapping them to
  fixed-sized vectors. These models are key to multimodal information retrieval
  and related tasks. However, CLIP models generally underperform in text-only
  tasks compared to specialized text models. This creates inefficiencies for
  information retrieval systems that keep separate embeddings and models for
  text-only and multimodal tasks. We propose a novel, multi-task contrastive
  training method to address this issue, which we use to train the jina-clip-v1
  model to achieve the state-of-the-art performance on both text-image and
  text-text retrieval tasks.
paper_authors._id: 66593db1d6898d357e15f98b
paper_authors.name: Andreas Koukounas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:43:26.776Z
paper_authors._id: 651e7084570ba4662812114b
paper_authors.avatarUrl: /avatars/1438e5caa483f63dc0da5ee7508ef7eb.svg
paper_authors.fullname: Andreas Koukounas
paper_authors.type: user
paper_authors.user: koukandre
paper_authors._id: 66593db1d6898d357e15f98c
paper_authors.name: Georgios Mastrapas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:43:34.731Z
paper_authors._id: 64f8620e492828088373ddf9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f8620e492828088373ddf9/g6XQmUzEPNMQYNc34gYpR.jpeg
paper_authors.fullname: Georgios Mastrapas
paper_authors.type: user
paper_authors.user: gmastrapas
paper_authors._id: 66593db1d6898d357e15f98d
paper_authors.name: Michael Günther
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:43:58.999Z
paper_authors._id: 6476ff2699a5ce743ccea3fc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6476ff2699a5ce743ccea3fc/zmFmF8tXXDaAGcl8RYiRr.jpeg
paper_authors.fullname: Michael Günther
paper_authors.type: user
paper_authors.user: michael-guenther
paper_authors._id: 66593db1d6898d357e15f98e
paper_authors.name: Bo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:24:56.501Z
paper_authors._id: 63491dc83d8dc83a55cb749c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63491dc83d8dc83a55cb749c/IoqJrOIaEnYO_S7si4KGp.jpeg
paper_authors.fullname: Bo Wang
paper_authors.type: user
paper_authors.user: bwang0911
paper_authors._id: 66593db1d6898d357e15f98f
paper_authors.name: Scott Martens
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:44:18.849Z
paper_authors._id: 64e4afc4ba1c93ea6d2a9335
paper_authors.avatarUrl: /avatars/27db123284c43bcfa6b8cb43da9ab35f.svg
paper_authors.fullname: Scott Martens
paper_authors.type: user
paper_authors.user: scott-martens
paper_authors._id: 66593db1d6898d357e15f990
paper_authors.name: Isabelle Mohr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:44:32.557Z
paper_authors._id: 64a830cd6cc1a9a131f62619
paper_authors.avatarUrl: /avatars/e7b0ad62e8fb8c99c8d75edf45688995.svg
paper_authors.fullname: Isabelle Mohr
paper_authors.type: user
paper_authors.user: isacat
paper_authors._id: 66593db1d6898d357e15f991
paper_authors.name: Saba Sturua
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:44:39.076Z
paper_authors._id: 64c23f6d569648a60737eddb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c23f6d569648a60737eddb/iZq7bp-yYaGl5VBVoN5Dg.jpeg
paper_authors.fullname: Saba Sturua
paper_authors.type: user
paper_authors.user: jupyterjazz
paper_authors._id: 66593db1d6898d357e15f992
paper_authors.name: Mohammad Kalim Akram
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:44:45.859Z
paper_authors._id: 64d22f33032a420d1863b6ea
paper_authors.avatarUrl: /avatars/ed3eaf4bab70dd6ab9a2b67b5928e4fb.svg
paper_authors.fullname: Mohammad Kalim Akram
paper_authors.type: user
paper_authors.user: makram93
paper_authors._id: 66593db1d6898d357e15f993
paper_authors.name: Joan Fontanals Martínez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:44:55.334Z
paper_authors._id: 65fb4368587b4439cd7a8dfa
paper_authors.avatarUrl: /avatars/ace63f3fb3fdcec9cd8d10f70e4569e7.svg
paper_authors.fullname: Joan Fontanals Martínez
paper_authors.type: user
paper_authors.user: JoanFM
paper_authors._id: 66593db1d6898d357e15f994
paper_authors.name: Saahil Ognawala
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:45:15.331Z
paper_authors._id: 655481c48344eaaaed1e49f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/655481c48344eaaaed1e49f4/Kve6ev1RnJ9-B5tfI0SWv.png
paper_authors.fullname: Saahil Ognawala
paper_authors.type: user
paper_authors.user: saahil-ognawala
paper_authors._id: 66593db1d6898d357e15f995
paper_authors.name: Susana Guzman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:45:21.688Z
paper_authors._id: 61a5f350995d6dd6541b00fd
paper_authors.avatarUrl: /avatars/9f90f008b10645e4d74e11838a9760ff.svg
paper_authors.fullname: Susana
paper_authors.type: user
paper_authors.user: Capricx
paper_authors._id: 66593db1d6898d357e15f996
paper_authors.name: Maximilian Werk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:45:27.913Z
paper_authors._id: 60638400b1703ddba0d458a7
paper_authors.avatarUrl: /avatars/50228a18e7f211275a09e3cbd6e2931e.svg
paper_authors.fullname: Maximilian Werk
paper_authors.type: user
paper_authors.user: mwerk
paper_authors._id: 66593db1d6898d357e15f997
paper_authors.name: Nan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:46:01.793Z
paper_authors._id: 60124e957bdbda85b0874f7d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60124e957bdbda85b0874f7d/tBxDFq00g3Gz8X0Bv2bwD.jpeg
paper_authors.fullname: Nan Wang
paper_authors.type: user
paper_authors.user: nan
paper_authors._id: 66593db1d6898d357e15f998
paper_authors.name: Han Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T09:46:16.569Z
paper_authors._id: 603763514de52ff951d89793
paper_authors.avatarUrl: /avatars/4484b361bb887fc73a6d38d953bfe9e0.svg
paper_authors.fullname: Han Xiao
paper_authors.type: user
paper_authors.user: hanxiao
paper_title: Jina CLIP: Your CLIP Model Is Also Your Text Retriever
paper_summary: Contrastive Language-Image Pretraining (CLIP) is widely used to train models
  to align images and texts in a common embedding space by mapping them to
  fixed-sized vectors. These models are key to multimodal information retrieval
  and related tasks. However, CLIP models generally underperform in text-only
  tasks compared to specialized text models. This creates inefficiencies for
  information retrieval systems that keep separate embeddings and models for
  text-only and multimodal tasks. We propose a novel, multi-task contrastive
  training method to address this issue, which we use to train the jina-clip-v1
  model to achieve the state-of-the-art performance on both text-image and
  text-text retrieval tasks.
paper_ai_summary: A novel multi-task contrastive training method improves CLIP model performance on both text-image and text-text retrieval tasks.
paper_ai_keywords: Contrastive Language-Image Pretraining CLIP embedding space multimodal information retrieval text-only tasks multi-task contrastive training jina-clip-v1