Two tokenizer initialization methods result in inconsistent segmentation results for special words
### System Info

transformers==4.17.0
torch==1.10.0
python==3.7.3

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction
```
# xlm-roberta-base directory: git clone https://huggingface.co/xlm-roberta-base
from transformers import XLMRobertaTokenizer
tokenizer_a = XLMRobertaTokenizer.from_pretrained('xlm-roberta-base/')
tokenizer_b = XLMRobertaTokenizer('xlm-roberta-base/sentencepiece.bpe.model')

t = 'texta<s>textb'
print(tokenizer_a.tokenize(t))
print(tokenizer_b.tokenize(t))
```

### Expected behavior

```
# what I expect is that both outputs: 
['▁text', 'a', '<s>', '▁text', 'b']
['▁text', 'a', '<s>', '▁text', 'b']

# However, in reality, their outputs are as follows:
['▁text', 'a', '<s>', '▁text', 'b']
['▁text', 'a', '<', 's', '>', 'text', 'b']
```


Why these two tokenizers have different segmentation results for special words?
Hey! Which version of transformers are you using? I tried to reproduce this but it did not work. Download the sentencepiece model from  the official repo, and it worked on main~<|||||>> Hey! Which version of transformers are you using? I tried to reproduce this but it did not work. Download the sentencepiece model from the official repo, and it worked on main~

my transformers version is 4.17.0.  <|||||>Ok, I can confirm that this is not working for me either! Thanks for reporting. This is most probably a problem related to `add_tokens` in the init: 
```python 
>>> tokenizer_a.unique_no_split_tokens
['</s>', '<mask>', '<pad>', '<s>', '<unk>']
>>> tokenizer_b.unique_no_split_tokens
```
<|||||>Linking this with #23909 as the core problem is similar: 
- Adding a special token does not recreate the Trie (when the special tokens are initialized)
- `from_pretrained`  calls `added_tokens = tokenizer.sanitize_special_tokens()`  which is when the special tokens are added to `unique_no_split`
- Initialising a model from a sentencepiece vocab file does not initialize the Trie (used to split the tokens), since the special tokens are not sanitized. <|||||>I've had to take on a few model addition, the fix PR will be updated soon! It requires a bit more refactoring than initially thought! 