Hi @rgwatwormhill, thanks for the answer!
Actually indeed that would make sense to need the  roberta vocabulary with the pre-trained weights, I was trying to do something similar to BioBERT (https://arxiv.org/ftp/arxiv/papers/1901/1901.08746.pdf 5) which seem to have used BERT weight initialization and a biomedical domain corpora…
I guess I’ll try both:

pre-train my domain specific with my tokenizer + from scratch
initialize training with roberta and continue training from there with my domain specific corpus processed with roberta tokenizer.

Still wondering how to initialize the weights to roberta’s though 
ps: yes using GCP credits and a paid Nvidia V100 for the simulation now, colab free GPU was not enough