Tokenizer becomes very slow after adding new tokens
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
@LysandreJik

## Information

Hi,

When I am trying to add a large number (50k) new tokens to BERT's tokenizer, the tokenizer becomes very slow, taking 29 seconds to tokenize a single short sentence.

## To reproduce

```
from transformers import BertTokenizer
import time

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

new_tokens = ["aaa"+str(i) for i in range(50000)]
tokenizer.add_tokens(new_tokens) # takes some time

sentence = "a short sentence."
start = time.time()
tokenizer.tokenize(sentence)
print(time.time() - start)

> 29.049
```



Hey @shauli-ravfogel, this should have been fixed on `master`. Can you try installing from source and let me know if it works better?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.