As far as I know, if you want to change the vocabulary in a significant way you’ll have to train the tokenizer from scratch, which also means doing the pretraining from scratch as well.
One thing I’ve seen in the docs before is the add_tokens method: Utilities for Tokenizers — transformers 4.3.0 documentation 2
Depending on how much you need to change the base vocabulary of the pretrained model, this might be a good start (although I’ve never tried this before so the results might be unexpected …)