Sorry for taking so long to reply. Your answer was very helpful to me and to my team. Thank you very much! We have trained RoBERTa from scratch on a portuguese corpora and plan to release it for the public eventually.