iamholmes:

from transformers import AutoModel
checkpoint = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
tokenizer.vocab



From a very quick scan, tokenizer.vocab is an attribute. tokenizer.get_vocab() is a function. They both return a dictionary with the same number of items. I’m not sure why we need both.