diiogo:

Dataset
We can use dataset from BrWaC (Brazilian Web as Corpus) a large Portuguese corpus or Wikidata in Portuguese.


The BrWaC is in the dataset hub of Hugging Face and OSCAR, too.
It might be interesting to mix the 2 together for training.
About the embeddings and weights initialization, starting from those of a trained RoBERTa model could  help, too

Download  the model from the HF Model Hub to get the weights.
Train a RoBERTa tokenizer on the training dataset to get the vocabulary.
Keep the ambeddings of the common tokens between the downloaded model vocabulary and the new one.
For the new tokens, initialize them.