Yes, these pre-trained models are trained for language tasks. So, if you have to use them, you’ll need to create your own vocabulary and tokenizer. A better idea may be to just train the transformer model yourself. Even if you go extract the pre-trained weights and load it into your model, I don’t think it will help a lot.
I remember I found some useful resources on the Google Magenta website.