It’s a great idea! Also, it would be nice to know how they distilled to get the multilingual CLIP 5.
Do they first distill the text encoder only and then align/fine-tune the multilingual embeddings with image embeddings? Maybe we can borrow some ideas for our CLIP training as well.