Okay, so obviously I’m not a Python guy… I see there’s some insanity in the language that allows class instances to be callable… (why, Python… WHY???)    …so I’m a bit stumped, but presumably it has to do with the fact that BertWordPieceTokenizer is not a subclass of PreTrainedTokenizer (which has the crazy attribute of being callable).
I’m really stuck. I’d just like to plug in my custom tokenizer, but it seems that when I hit “LineByLineTextDataset”, I’m going to hit the same callable error. I tried running with the default tokenization and although my vocab went down from 1073 to 399 tokens, my sequence length went from 128 to 833 tokens. Hence the desire to load my tokenizer from the hand-written vocab.
Aack!
UPDATE: Okay, I hadn’t realized I could do it with BertTokenizerFast. I haven’t totally verified that this is working, but so far it looks correct.