Hi @snugger I thought the original tokenizer just skips the words in list_of_words if tokenizer already has these words in it.
Is it necessary to pre-process this list_of_words before using the add_tokens method so that list_of_words consists of only new words?