XLNET tokenization changes after saving and loading
## Environment info
- Fresh Colab env with !pip install transformers[sentencepiece]
- `transformers` version: 4.5.0 
- Behavior also observed in previous Transformers versions >= 4.3.0

### Who can help
xlnet tokenization: @patrickvonplaten

## Information
Our CI reported different tokenization when saving and loading the xlnet tokenizer with added tokens:
```python
Before loading: ['▁Some', '▁Text', '▁with', 'neverseentokens', '▁and', '▁more', '<sep>', '<cls>']
After loading: ['▁Some', '▁Text', '▁with', '▁', 'neverseentokens', '▁and', '▁more', '<sep>', '<cls>']
```
## To reproduce

```python
from transformers import AutoTokenizer, AutoConfig

def test_tokenizers():
    basic_text = "Some Text with neverseentokens and more"

    tokenizer_orig = AutoTokenizer.from_pretrained("xlnet-base-cased")
    config_orig = AutoConfig.from_pretrained("xlnet-base-cased")
    tokenizer_orig.add_tokens(new_tokens=["neverseentokens"])
    tokenizer_orig.save_pretrained("testsave")
    config_orig.save_pretrained("testsave")
    tokens_before = tokenizer_orig.encode_plus(basic_text).encodings[0].tokens
    
    tokenizer_loaded = AutoTokenizer.from_pretrained("testsave")
    tokens_after = tokenizer_loaded.encode_plus(basic_text).encodings[0].tokens

    print(f"Before loading: {tokens_before}")
    print(f"After loading: {tokens_after}")
    assert tokens_before == tokens_after

if __name__ == "__main__":
    test_tokenizers()
```

## Expected behavior

Same tokenization before and after loading.

Wow, that was quick, I just saw @stefan-it recent comment in our failing PR: https://github.com/deepset-ai/FARM/pull/727#issuecomment-814641885

So loading the XLNet Tokenizer in slow mode with
```python
tokenizer_orig = AutoTokenizer.from_pretrained("xlnet-base-cased", use_fast=True, from_slow=True)
```

resolved the issue.<|||||>Thanks for opening an issue @Timoeller. The `from_slow=True` is a bandaid over the underlying issue.

We're actively working on solving this issue, but it will take a bit of time as it's hard to do in a backwards compatible way.