Pataleros:

=> Well, isn’t it exactly what we do when we run tokenizer.add(['word1', 'word2']) and then model.resize_token_embeddings(len(tokenizer)) ? Only update the shape of last layer ?


I don’t remember any of those arcane function  either they didn’t exist a year ago or I did a poor job searching for them. I have to give them a try !



 Pataleros:

=> As Bert is using a WordPiece tokenization (instead of Roberta BPE), why didn’t you / couldn’t we use a SpaCY WordPiece tokenization before adding to the vocabulary ? I found this example (see Annex at the bottom) where this is what they do. I am considering switching from Roberta to Bert for this ?


I definitly chose Bert because the tokenizing part was easier for me