Sorry, I was a bit too quick.
What is your problem with just training your custom tokenizer and using it?
I don’t understand what you mean with this:

The tokenizers provided by the transformers library are all supposed to be pretrained. For example, one issue is the way the subwords are handled ("##" in BERT tokenizer, which is handled differently in tokenizers).

It seems that this 10 is exactly what you are looking for. Once trained you can just load the tokenizer with tokenizers rather than with transformers.PreTrainedTokenizer.