❓Adding new tokens to pre-trained tokenizer
## Details
Hi, I am working with DistilBERT multilingual model for sequence classification tasks where I need to add some additional languages apart from mentioned [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages). And for that, I am struggling to find the correct way to update tokenizer. From the documentation, I inferred that first, i have to get all new tokens in a list, call `tokenizer.add_tokens()` and then again i have to pass those new sentences to tokenizer and get them tokenized. So the real question: is there any method which i use to update tokenizer and tokenize sentence at the same time (when tokenizer sees unknown token it adds the token to the dictionary). Thanks in advance.
docs are pretty nice imho;<|||||>If I had found what I looking for in the documentation then why would I open an issue and waste someone else's time? I know other approaches to this problem but one seemed to be more time saving so just checking for implementation available. But now it seems that people are more concerned about critical bugs only in the issue section. Closing the issue for good.<|||||>There is no way to dynamically add unknown tokens to the vocabulary. The simplest way to do it would be to encode the sequence, detect unknowns, and then add these to the vocabulary, which seems to be what you did!

Please be aware that you will have to resize the model's embedding matrix according to the tokens you've added.<|||||>Or you can map all such tokens a group of them maybe to an OOV kinda token as well. <|||||>So the only way to update tokenizer is to get all unknowns first then resize model embedding matrix. Thanks @LysandreJik and @AdityaSoni19031997 