I also meet this problem and I get my script run by manually checking the vocabulary size the tokenizer has and change the vocab
_size in the configuration of the model.
I think changing vocab_size argument in the tokenizer.train() function doesn’t change anything regarding the vocabulary size.