Collection of Tokenizer issues
### System Info

```shell
Transformers + Tokenizers
```


### Who can help?

This Issue is a summary of multiple problems that we are currently encountering with Tokenizers. To solve them we'll need a more profound discussion of:
- To what extend fast and slow tokenizers should be aligned
- Whether all slow tokenizers should be kept
- How to treat special tokens
- Whether all internal methods of tokenizer should be exposed

Relevant issues/PRs:
https://github.com/huggingface/transformers/issues/15420
https://github.com/huggingface/transformers/issues/16336
https://github.com/huggingface/transformers/issues/16334
https://github.com/huggingface/transformers/issues/16337
https://github.com/huggingface/transformers/issues/15138
https://github.com/huggingface/transformers/issues/16339
https://github.com/huggingface/transformers/pull/15775

To community:
At the moment we sadly don't find the time to dive deeper here, but we're trying hard to allocate time to discuss the strategy here soon.



### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See issues above

### Expected behavior

```shell
Don't know yet
```

cc @LysandreJik @sgugger @SaulLu @Narsil @patil-suraj <|||||>Internal thread: https://huggingface.slack.com/archives/C01N44FJDHT/p1647966224411599<|||||>Another one: https://github.com/huggingface/transformers/issues/16787#issuecomment-1100009727<|||||>It would be nice to add those to a project so that we may track the resolution of these issues.<|||||>Another one: https://github.com/huggingface/transformers/issues/16225
<|||||>Another one: https://github.com/huggingface/transformers/issues/17595<|||||>https://github.com/huggingface/tokenizers/issues/1011