RobertaTokenizer runs slowly after add _tokens
## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I use RobertaTokenizer like this:  
```python
tokenizer = RobertaTokenizer.from_pretrained(FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
tokenizer.add_tokens([x.strip() for x in open('add_tokens.txt').readlines()])
```  
There are about 200 words in `add_tokens.txt`. I tested it on 300 sample datasets, and 250% more time after using `add_tokens.txt`. Is there any way to optimize it？
Hi, I've done a short study and I confirm the behavior you see.
I've proposed a simple PR attached that gives interesting results and quite important speed improvement in any case.
To be discussed!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
