I’ve been looking into how the tokenizer gets loaded. I’ve got a vocab.json and a merges.txt. I added a print statement to run_mlm.py right here: transformers/run_mlm.py at 49bee0aea44ef29c08d48f818f356275ef223da8 · huggingface/transformers · GitHub 5
And I see that the autotokenizer sets the model_max_length to an incredibly large number: model_max_len=1000000000000000019884624838656
06/08/2021 09:07:05 - INFO - __main__ -   tokenizer: PreTrainedTokenizerFast(name_or_path='/home/cleong/projects/personal/colin-summer-2021/EsperBERTo/', vocab_size=52000, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', special_tokens={'bos_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'eos_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'unk_token': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'sep_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'pad_token': AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'cls_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=True)})