BertTokenizer slowly on latest versions
### System Info

Hi, I test the execution time of my program when adopt different transformers version.
In my program, when the transformers version was **4.21.3**, the execution time was **7.12s**, but when I changed the lightgbm to the **4.3.0**, the execution tim was **2.79s**.
I record the system info on different versions.
[4.21.3.txt](https://github.com/huggingface/transformers/files/9532526/4.21.3.txt)
[4.10.0.txt](https://github.com/huggingface/transformers/files/9532528/4.10.0.txt)
[4.3.0.txt](https://github.com/huggingface/transformers/files/9532527/4.3.0.txt)

### Who can help?

@LysandreJik 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1: Code
```python

from transformers import BertTokenizer
import pandas as pd
import time
start = time.time()
train_df = pd.read_csv('train.csv')
train_df.head()
tweets = train_df['text'].values
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')
def encode_sentence(s):
    tokens = list(tokenizer.tokenize(s))
    tokens.append('[SEP]')
    return tokenizer.convert_tokens_to_ids(tokens)
max_len = 50
x_train = []
for tweet in tweets:
    vec = encode_sentence(tweet)
    x_train.append(vec[:max_len] + [0] * (max_len - len(vec)))
end = time.time()
print("Time:", end-start)

```
2: Dataset
[train.csv](https://github.com/huggingface/transformers/files/9532537/train.csv)


### Expected behavior

The execution time are same on different versions, or the latest version is better than older versions.
|Version|Execution time|
|--|--|
|4.21.3|7.12s|
|4.10.0|8.85s|
|4.3.0|2.79s|
cc @SaulLu <|||||>Hi @PerformanceDetect ,

Thanks for sharing this benchmark with us! Indeed it would be interesting to find out what addition caused this slowdown, do you think you would have some time to investigate further?

Otherwise, if your usage is speed sensitive, I recommend you to use the fast version of the tokenizer :hugs: <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.