I wonder if something is going on with the tokenizer…
When running on the 4.5.1 version, I changed in run_mlm script AutoTokenizer to BertTokenizer (it had a bug perhaps, wouldn’t recognize my tokenizer).
On 4.9.2 I get this error when running the script:
“The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is ‘T5TokenizerFast’.
The class this function is called from is ‘BertTokenizer’.”
That’s the code I’m using to train the tokenizer:
tokenizer = BertWordPieceTokenizer()
tokenizer.train(files=[‘wikipedia_1e6.txt], vocab_size=30000)
tokenizer.save_model(’/tokenizers/vocab_folder’)
tokenizer = BertTokenizer.from_pretrained(’/tokenizers/vocab_folder’)
tokenizer.save_pretrained(’/tokenizers/wikipedia_1e6/’)
However, when training tokenizer with transformers 4.9.2 and using it in run_mlm I still get a bigger train loss (than 4.5.1), but no warning.