patrickvonplaten:

It would be important to see if the data can be used and if so how!


I think this initiative can qualify as “lawful use in scientific research”, so I don’t think there is any problem. In any case, I can communicate directly with the license owners and ask them about it.

Also maybe it might make sense to fine-tune the official CLIP weights on the medical data instead of pretraining from scratch?

Considering the amount of data available, fine-tuning will probably work better. The main reason why I proposed to train it from scratch instead of finetuning is the vocabulary. Since the medical vocabulary stands out for its large number of unusual terms in more standard domains with which standard tokenizers tend to have problems (See for instance Beltagy et al., (2019)).
I suppose that techniques such as recycling (de Vries & Nissim, 2021) or Adapters (Houlsby et al., 2019; Pfeiffer et al., 2020), could solve this problem.