hey @anon58275033 you can tokenize the corpus in the usual way after you’ve added new tokens with tokenizer.add_tokens. since it seems you’re doing masked language modeling, you might want to check out this tutorial to see how this is done: Google Colaboratory 43