Okay. I don’t have time to run more than 1 epoch today, so maybe I can try tomorrow.
Nevertheless, I agree with your belief that the WARNING is not important , since all of the warned layers seem to be output-head of the PretrainingRoberta, while we are not using that head in finetuning.
I also think version should be irrelevant since Roberta is a relatively old model in HF and has been very stable.
I still suggest you to use TPU though since it takes too much time to experiment more than 1 epoch.