Slow decoding with many special tokens in vocabulary
### System Info

present across multiple versions

### Who can help?

@younesbelkada 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import T5Tokenizer
from time import time
from random import randint

t1 = T5Tokenizer.from_pretrained('t5-base')
t2 = T5Tokenizer.from_pretrained('t5-base', extra_ids=2000)
to_decode = [randint(0, 32000) for i in range(10000)]

start = time()
t1.decode(to_decode)
print("few special tokens:", time() - start)

start = time()
t2.decode(to_decode)
print("many special tokens:", time() - start)
```

### Expected behavior

The slowdown should not be so drastic. The cause is an inefficient implementation of [`all_special_ids`](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1293) and [`all_special_tokens`](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1267). Additionally, generating them on the fly incurs a large overhead since this attribute is queried for every id to be decoded ([here](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils.py#L907) and [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/tokenization_t5.py#L324)).
Slow tokenizers are... slow. That's why we wrote the tokenizers library ;-) Why not use `T5TokenzierFast` which doesn't have the same problem?<|||||>`T5TokenzierFast` does not have byte-fallback + why artificially handicap the slow tokenizer if it could be more efficient (using sets instead of lists and computing the attribute only when it's updated)?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.