Thank you very much for your helpful guide, it works fine after 6-7 epochs. I think this delay in learning is due to the mentioned warning because I used TFAutoModel.from_pretrained(‘roberta-base’) instead of TFRobertaModel.from_pretrained(‘roberta-base’), and as it initializes all of the weights, it converges much sooner than the existing model.