Hi! The project sounds great. I already implemented a CLIP like model with Pytorch Lightning / Timm and transformers (Universal Sentence Encoder and RoBERTa). Looking forward to work with CLIP, JAX/Flax and TPUs. So count me in 
Florian