As Patrick said, please see if the data can be made available before the sprint!
And regarding fine-tuning and medical vocabulary I think in this case we could maybe use a text encoder which is trained on medical data and then pair it with CLIP’s vision model instead of starting from scratch.