Tokenizers became slow compared to 2.8.0
## Environment info

- `transformers` version: 3.1.0
- Platform: Ubuntu 20.04
- Python version: 3.7.9
- PyTorch version (GPU?): No
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help 
 tokenizers: @mfuntowicz

## To reproduce

Steps to reproduce the behavior:

```python
import timeit
import numpy as np
from transformers import __version__ as trans_version
from transformers import (
    CTRLTokenizer,
    GPT2Tokenizer,
    RobertaTokenizer,
    XLMTokenizer,
    XLNetTokenizer
)

Tok_CLASSES = {
    "gpt2": GPT2Tokenizer,
    "ctrl": CTRLTokenizer,
    "roberta-large": RobertaTokenizer,
    "xlnet-large-cased": XLNetTokenizer,
    "xlm-mlm-100-1280": XLMTokenizer,
}

print(trans_version)
for k, v in Tok_CLASSES.items():
    tokenizer_class = v
    tokenizer = tokenizer_class.from_pretrained(k)

    text = """<|endoftext|>🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides
                general-purpose
                architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU)
                and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep
                interoperability between TensorFlow 2.0 and PyTorch.</s> <eos>"""

    print(tokenizer.__class__)
    r = timeit.repeat(stmt="tokenizer.encode(text)", repeat=100, number=500, globals=globals())
    rounding = 4
    print('Time taken (mean ± 3std):',
          str(np.round(np.mean(r), rounding)) + '±' + str(np.round(3 * np.std(r), rounding)))

```
In 3.1.0 output is:
```
3.1.0
<class 'transformers.tokenization_gpt2.GPT2Tokenizer'>
Time taken (mean ± 3std): 0.1808±0.0115
<class 'transformers.tokenization_ctrl.CTRLTokenizer'>
Time taken (mean ± 3std): 0.0678±0.0015
<class 'transformers.tokenization_roberta.RobertaTokenizer'>
Time taken (mean ± 3std): 0.2051±0.0024
<class 'transformers.tokenization_xlnet.XLNetTokenizer'>
Time taken (mean ± 3std): 0.1567±0.002
<class 'transformers.tokenization_xlm.XLMTokenizer'>
Time taken (mean ± 3std): 0.3601±0.0248
```

## Expected behavior

In 2.8.0 output is (and i hope, even these times can be improved without using Fast versions). 
```
<class 'transformers.tokenization_gpt2.GPT2Tokenizer'>
Time taken (mean ± 3std): 0.1808±0.0115
<class 'transformers.tokenization_ctrl.CTRLTokenizer'>
Time taken (mean ± 3std): 0.0678±0.0015
<class 'transformers.tokenization_roberta.RobertaTokenizer'>
Time taken (mean ± 3std): 0.2051±0.0024
<class 'transformers.tokenization_xlnet.XLNetTokenizer'>
Time taken (mean ± 3std): 0.1567±0.002
<class 'transformers.tokenization_xlm.XLMTokenizer'>
Time taken (mean ± 3std): 0.3601±0.0248
```
## TLDR
With GPT2 and CTRL 3.1.0 tokenizer.encode (with default options) takes ~1.3x time compared to 2.8.0 code. I think this can be improved|solved if code like https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L254-L256 was not executed every tokenization call but for example once tokens added (and result stored in self.all_special_tokens_extended). Maybe this is not only place with unnecessary calculations per tokenization.
For example `self.encoder.get(self.unk_token)` from https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_gpt2.py#L244 could be stored once in some property updated with unk_token change to reduce time to call it every time token-to-id conversion happens.
Same storage idea for https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L266

Working with special tokens attributes also became slower:
```python
import timeit
import numpy as np
from transformers import __version__ as trans_version
from transformers import (
    CTRLTokenizer,
    GPT2Tokenizer,
    RobertaTokenizer,
    XLMTokenizer,
    XLNetTokenizer
)

Tok_CLASSES = {
    "gpt2": GPT2Tokenizer,
    "ctrl": CTRLTokenizer,
    "roberta-large": RobertaTokenizer,
    "xlnet-large-cased": XLNetTokenizer,
    "xlm-mlm-100-1280": XLMTokenizer,
}

print(trans_version)
rounding = 4
for k, v in Tok_CLASSES.items():
    tokenizer_class = v
    tokenizer = tokenizer_class.from_pretrained(k, verbose=False)

    print(tokenizer.__class__)
    # print(tokenizer.encode(text))
    if tokenizer.eos_token is not None:
        r_eos_token = timeit.repeat(stmt="tokenizer.eos_token", repeat=100, number=100000, globals=globals())
        r_eos_token_id = timeit.repeat(stmt="tokenizer.eos_token_id", repeat=100, number=100000, globals=globals())
        print('Get eos_token, time taken (mean ± 3std):',
              str(np.round(np.mean(r_eos_token), rounding)) + '±' + str(np.round(3 * np.std(r_eos_token), rounding)))
        print('Get eos_token_id, time taken (mean ± 3std):',
              str(np.round(np.mean(r_eos_token_id), rounding)) + '±' + str(
                  np.round(3 * np.std(r_eos_token_id), rounding)))
    if tokenizer.bos_token is not None:
        r_bos_token = timeit.repeat(stmt="tokenizer.bos_token", repeat=100, number=100000, globals=globals())
        r_bos_token_id = timeit.repeat(stmt="tokenizer.bos_token_id", repeat=100, number=100000, globals=globals())
        print('Get bos_token, time taken (mean ± 3std):',
              str(np.round(np.mean(r_bos_token), rounding)) + '±' + str(np.round(3 * np.std(r_bos_token), rounding)))
        print('Get bos_token_id, time taken (mean ± 3std):',
              str(np.round(np.mean(r_bos_token_id), rounding)) + '±' + str(
                  np.round(3 * np.std(r_bos_token_id), rounding)))
    if tokenizer.unk_token is not None:
        r_unk_token = timeit.repeat(stmt="tokenizer.unk_token", repeat=100, number=100000, globals=globals())
        r_unk_token_id = timeit.repeat(stmt="tokenizer.unk_token_id", repeat=100, number=100000, globals=globals())
        print('Get unk_token, time taken (mean ± 3std):',
              str(np.round(np.mean(r_unk_token), rounding)) + '±' + str(np.round(3 * np.std(r_unk_token), rounding)))
        print('Get unk_token_id, time taken (mean ± 3std):',
              str(np.round(np.mean(r_unk_token_id), rounding)) + '±' + str(
                  np.round(3 * np.std(r_unk_token_id), rounding)))
```
gives for 2.8.0
```
2.8.0
<class 'transformers.tokenization_gpt2.GPT2Tokenizer'>
Get eos_token, time taken (mean ± 3std): 0.0104±0.0003
Get eos_token_id, time taken (mean ± 3std): 0.0715±0.0059
Get bos_token, time taken (mean ± 3std): 0.0101±0.0021
Get bos_token_id, time taken (mean ± 3std): 0.0687±0.0155
Get unk_token, time taken (mean ± 3std): 0.0101±0.0001
Get unk_token_id, time taken (mean ± 3std): 0.0632±0.0004
<class 'transformers.tokenization_ctrl.CTRLTokenizer'>
Get unk_token, time taken (mean ± 3std): 0.0098±0.0002
Get unk_token_id, time taken (mean ± 3std): 0.0639±0.0008
<class 'transformers.tokenization_roberta.RobertaTokenizer'>
Get eos_token, time taken (mean ± 3std): 0.0099±0.0003
Get eos_token_id, time taken (mean ± 3std): 0.0644±0.0017
Get bos_token, time taken (mean ± 3std): 0.0093±0.0001
Get bos_token_id, time taken (mean ± 3std): 0.064±0.0003
Get unk_token, time taken (mean ± 3std): 0.0094±0.0002
Get unk_token_id, time taken (mean ± 3std): 0.0727±0.0039
<class 'transformers.tokenization_xlnet.XLNetTokenizer'>
Get eos_token, time taken (mean ± 3std): 0.01±0.0002
Get eos_token_id, time taken (mean ± 3std): 0.0848±0.0021
Get bos_token, time taken (mean ± 3std): 0.0104±0.0003
Get bos_token_id, time taken (mean ± 3std): 0.0847±0.0072
Get unk_token, time taken (mean ± 3std): 0.0097±0.0001
Get unk_token_id, time taken (mean ± 3std): 0.084±0.0007
<class 'transformers.tokenization_xlm.XLMTokenizer'>
Get bos_token, time taken (mean ± 3std): 0.01±0.0001
Get bos_token_id, time taken (mean ± 3std): 0.0646±0.001
Get unk_token, time taken (mean ± 3std): 0.0098±0.0001
Get unk_token_id, time taken (mean ± 3std): 0.0639±0.0004
```
and for 3.1.0 (2x...4x slower):
```
3.1.0
<class 'transformers.tokenization_gpt2.GPT2Tokenizer'>
Get eos_token, time taken (mean ± 3std): 0.0422±0.0004
Get eos_token_id, time taken (mean ± 3std): 0.1465±0.0015
Get bos_token, time taken (mean ± 3std): 0.0418±0.0005
Get bos_token_id, time taken (mean ± 3std): 0.1453±0.0009
Get unk_token, time taken (mean ± 3std): 0.0417±0.0002
Get unk_token_id, time taken (mean ± 3std): 0.1519±0.0186
<class 'transformers.tokenization_ctrl.CTRLTokenizer'>
Get unk_token, time taken (mean ± 3std): 0.0163±0.0003
Get unk_token_id, time taken (mean ± 3std): 0.0821±0.0006
<class 'transformers.tokenization_roberta.RobertaTokenizer'>
Get eos_token, time taken (mean ± 3std): 0.0419±0.0029
Get eos_token_id, time taken (mean ± 3std): 0.1462±0.004
Get bos_token, time taken (mean ± 3std): 0.042±0.0004
Get bos_token_id, time taken (mean ± 3std): 0.1544±0.0311
Get unk_token, time taken (mean ± 3std): 0.0449±0.0016
Get unk_token_id, time taken (mean ± 3std): 0.1511±0.006
<class 'transformers.tokenization_xlnet.XLNetTokenizer'>
Get eos_token, time taken (mean ± 3std): 0.0165±0.0004
Get eos_token_id, time taken (mean ± 3std): 0.0918±0.0043
Get bos_token, time taken (mean ± 3std): 0.0164±0.0003
Get bos_token_id, time taken (mean ± 3std): 0.0931±0.0034
Get unk_token, time taken (mean ± 3std): 0.0166±0.0002
Get unk_token_id, time taken (mean ± 3std): 0.0933±0.0004
<class 'transformers.tokenization_xlm.XLMTokenizer'>
Get bos_token, time taken (mean ± 3std): 0.0162±0.0003
Get bos_token_id, time taken (mean ± 3std): 0.0801±0.0008
Get unk_token, time taken (mean ± 3std): 0.016±0.0003
Get unk_token_id, time taken (mean ± 3std): 0.0827±0.0002
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
