AlbertTokenizer handles special tokens incorrectly
If I download the pretrained vocab `https://huggingface.co/albert-base-v1/resolve/main/spiece.model` to the local file system and use the following snippet, the tokenizer does not handle the special tokens properly:
```
tokenizer = AlbertTokenizer('spiece.model')
tokenizer.tokenize('[CLS] Hello World ! [SEP]')
['▁[', 'cl', 's', ']', '▁hello', '▁world', '▁', '!', '▁[', 's', 'ep', ']']
```
If I use `from_pretrained` to load the vocab, it works well:
```
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v1')
tokenizer.tokenize('[CLS] Hello World ! [SEP]')
['[CLS]', '▁hello', '▁world', '▁', '!', '[SEP]']
```
I'm taking a look at why this is, it seems that we have a differing behavior between `from_pretrained` and the initialization method. I tried loading that `spiece.model` directly with `from_pretrained` and it behaves normally, and so does pointing to a directory containing solely that file.

I'm taking a look and will come back to you.<|||||>@LysandreJik the problem is that `unique_no_split_tokens` is not initialised when you create a tokenizer from `__init__`.

See: https://stackoverflow.com/questions/64631665/what-is-the-difference-in-robertatokenizer-and-from-pretrained-way-of-initia/64640570#64640570<|||||>Indeed, thanks for investigating @cronoik. Do you want to open a PR with a fix?<|||||>Yes, I can but I would like to discuss this before because it affects the core of the library and all tokenizers (I have only checked the slow tokenizers yet, but it probably applies to the fast tokenizers as well.). 

When a user calls `.from_pretrained`, the tokenizer is created with `__init__` in the `._from_pretrained` method of the `PreTrainedTokenizerBase` class ([line 1868](https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/tokenization_utils_base.py#L1868)). The problem is now, that `._from_pretrained` does some magic from line [1881](https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/tokenization_utils_base.py#L1881) to [1909](https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/tokenization_utils_base.py#L1909), that is not executed when you create the tokenizer from `__init__` directly. 

So, simply said all I need to do is to move this magic to the `__init__` method and remove it from the `._from_pretrained`?

@LysandreJik <|||||>Pinging @thomwolf for advice on tokenizers loading methods.<|||||>This issue has been stale for 1 month.<|||||>@LysandreJik Maybe the [documentation](https://huggingface.co/transformers/main_classes/tokenizer.html) should be updated to at least tell the people that the recommended way to initialize a tokenizer is `from_pretrained` and that is not guaranteed that `__init__` will work properly?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>It is closed without solving the issue?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Pinging @SaulLu here, as I also encountered this.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.