@gsarti I see, but still I’m not sure how that is adopted in CLIP.
Perhaps, they use the pre-trained CLIP text encoder as the teacher and a bilingual student encoder. Next, they adopt the trained student as the CLIP text encoder.
But now, IMO the multilingual distillation has misaligned text and image embeddings in CLIP’s contrastive space (is that any right?).
I was wondering if they run an additional contrastive-loss pretraining with the new encoder.
I hope we’ll have time to discuss that and more on a dedicated Discord/Slack server if the project gets funded  (or even if it doesn’t)
Totally agree on MSCOCO-it. Together with WIT (it), I think they’re a good starting point.