Jina CLIP: Your CLIP Model Is Also Your Text Retriever
Contrastive Language-Image Pretraining (CLIP) is widely used to train models
to align images and texts in a common embedding space by mapping them to
fixed-sized vectors. These models are key to multimodal information retrieval
and related tasks. However, CLIP models generally underperform in text-only
tasks compared to specialized text models. This creates inefficiencies for
information retrieval systems that keep separate embeddings and models for
text-only and multimodal tasks. We propose a novel, multi-task contrastive
training method to address this issue, which we use to train the jina-clip-v1
model to achieve the state-of-the-art performance on both text-image and
text-text retrieval tasks.
66593db1d6898d357e15f98b
Andreas Koukounas
admin_assigned
2024-05-31T09:43:26.776Z
651e7084570ba4662812114b
/avatars/1438e5caa483f63dc0da5ee7508ef7eb.svg
Andreas Koukounas
user
koukandre
66593db1d6898d357e15f98c
Georgios Mastrapas
admin_assigned
2024-05-31T09:43:34.731Z
64f8620e492828088373ddf9
https://cdn-avatars.huggingface.co/v1/production/uploads/64f8620e492828088373ddf9/g6XQmUzEPNMQYNc34gYpR.jpeg
Georgios Mastrapas
user
gmastrapas
66593db1d6898d357e15f98d
Michael Günther
admin_assigned
2024-05-31T09:43:58.999Z
6476ff2699a5ce743ccea3fc
https://cdn-avatars.huggingface.co/v1/production/uploads/6476ff2699a5ce743ccea3fc/zmFmF8tXXDaAGcl8RYiRr.jpeg
Michael Günther
user
michael-guenther
66593db1d6898d357e15f98e
Bo Wang
claimed_verified
2024-05-31T06:24:56.501Z
63491dc83d8dc83a55cb749c
https://cdn-avatars.huggingface.co/v1/production/uploads/63491dc83d8dc83a55cb749c/IoqJrOIaEnYO_S7si4KGp.jpeg
Bo Wang
user
bwang0911
66593db1d6898d357e15f98f
Scott Martens
admin_assigned
2024-05-31T09:44:18.849Z
64e4afc4ba1c93ea6d2a9335
/avatars/27db123284c43bcfa6b8cb43da9ab35f.svg
Scott Martens
user
scott-martens
66593db1d6898d357e15f990
Isabelle Mohr
admin_assigned
2024-05-31T09:44:32.557Z
64a830cd6cc1a9a131f62619
/avatars/0c6ba301a66f2db73049c9fe0e97f2ef.svg
Isabelle Mohr
user
isacat
66593db1d6898d357e15f991
Saba Sturua
admin_assigned
2024-05-31T09:44:39.076Z
64c23f6d569648a60737eddb
https://cdn-avatars.huggingface.co/v1/production/uploads/64c23f6d569648a60737eddb/iZq7bp-yYaGl5VBVoN5Dg.jpeg
Saba Sturua
user
jupyterjazz
66593db1d6898d357e15f992
Mohammad Kalim Akram
admin_assigned
2024-05-31T09:44:45.859Z
64d22f33032a420d1863b6ea
/avatars/ed3eaf4bab70dd6ab9a2b67b5928e4fb.svg
Mohammad Kalim Akram
user
makram93
66593db1d6898d357e15f993
Joan Fontanals Martínez
admin_assigned
2024-05-31T09:44:55.334Z
65fb4368587b4439cd7a8dfa
/avatars/ace63f3fb3fdcec9cd8d10f70e4569e7.svg
Joan Fontanals Martínez
user
JoanFM
66593db1d6898d357e15f994
Saahil Ognawala
admin_assigned
2024-05-31T09:45:15.331Z
655481c48344eaaaed1e49f4
https://cdn-avatars.huggingface.co/v1/production/uploads/655481c48344eaaaed1e49f4/Kve6ev1RnJ9-B5tfI0SWv.png
Saahil Ognawala
user
saahil-ognawala
66593db1d6898d357e15f995
Susana Guzman
admin_assigned
2024-05-31T09:45:21.688Z
61a5f350995d6dd6541b00fd
/avatars/9f90f008b10645e4d74e11838a9760ff.svg
Susana
user
Capricx
66593db1d6898d357e15f996
Maximilian Werk
admin_assigned
2024-05-31T09:45:27.913Z
60638400b1703ddba0d458a7
/avatars/50228a18e7f211275a09e3cbd6e2931e.svg
Maximilian Werk
user
mwerk
66593db1d6898d357e15f997
Nan Wang
admin_assigned
2024-05-31T09:46:01.793Z
60124e957bdbda85b0874f7d
https://cdn-avatars.huggingface.co/v1/production/uploads/60124e957bdbda85b0874f7d/tBxDFq00g3Gz8X0Bv2bwD.jpeg
Nan Wang
user
nan
66593db1d6898d357e15f998
Han Xiao
admin_assigned
2024-05-31T09:46:16.569Z
603763514de52ff951d89793
https://cdn-avatars.huggingface.co/v1/production/uploads/603763514de52ff951d89793/n-QouGYg7oE5QeDaAb3Ns.png
Han Xiao
user
hanxiao
Jina CLIP: Your CLIP Model Is Also Your Text Retriever
Contrastive Language-Image Pretraining (CLIP) is widely used to train models
to align images and texts in a common embedding space by mapping them to
fixed-sized vectors. These models are key to multimodal information retrieval
and related tasks. However, CLIP models generally underperform in text-only
tasks compared to specialized text models. This creates inefficiencies for
information retrieval systems that keep separate embeddings and models for
text-only and multimodal tasks. We propose a novel, multi-task contrastive
training method to address this issue, which we use to train the jina-clip-v1
model to achieve the state-of-the-art performance on both text-image and
text-text retrieval tasks.
A novel multi-task contrastive training method improves CLIP model performance on both text-image and text-text retrieval tasks.
Contrastive Language-Image Pretraining CLIP embedding space multimodal information retrieval text-only tasks multi-task contrastive training jina-clip-v1