Thanks for your suggestions. Using ANN has been on my mind from the beginning. After looking for a few other options I also saw NGT which seems to be producing the best results too.
For within image domain pre-training, I would rather prefer the recent self-supervised methods like DINO because they are well formulated and we’ll established. DINO is also particularly good at this task.
Just as an FYI, one of the Keras links above (shared by @Bhack) actually shows you how to implement a CLIP like model minimally. If you haven’t, definitely check that out