Thanks for the information @lesscomfortable - in that case I wonder whether it would be faster / cheaper to just fine-tune an existing Spanish language model like this one on your corpus: dccuchile/bert-base-spanish-wwm-uncased · Hugging Face 2
(There’s already a script for doing this here: transformers/examples/language-modeling at master · huggingface/transformers · GitHub 15)
By all means pretrain from scratch if you have enough text data / compute, but the above suggestion would serve as a quick benchmark to compare against 
PS. I realise that there are quite some differences between Spanish from Argentina vs Spain, and I’m not sure what the Spanish BERT model was trained on …