Creating XLNetTokenizer from Custom ByteLevelBPETokenizer Throws OSError
### System Info

- `transformers` version: 4.28.1
- Platform: Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @ArthurZucker @younesbelkada

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hello,

I'm running into some issues using a custom tokenizer with XLNet. I have a ByteLevelBPETokenizer (located in `./tokenizer`) that I already trained, but when trying to load it with XLNetTokenizer, I get an OSError.

```
>>> from transformers import XLNetTokenizer
>>> tokenizer = XLNetTokenizer.from_pretrained("tokenizer", local_files_only=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hiekense/.local/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1795, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for 'tokenizer'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'tokenizer' is the correct path to a directory containing all relevant files for a XLNetTokenizer tokenizer.
```

I've read in a few places that it's likely due to a missing `tokenizer_config.json`, so I tried dropping in the default one from `xlnet-base-cased`.

```
{
  "additional_special_tokens": [
    "<eop>",
    "<eod>"
  ],
  "bos_token": "<s>",
  "clean_up_tokenization_spaces": true,
  "cls_token": "<cls>",
  "do_lower_case": false,
  "eos_token": "</s>",
  "keep_accents": false,
  "mask_token": {
    "__type": "AddedToken",
    "content": "<mask>",
    "lstrip": true,
    "normalized": true,
    "rstrip": false,
    "single_word": false
  },
  "model_max_length": 1000000000000000019884624838656,
  "pad_token": "<pad>",
  "remove_space": true,
  "sep_token": "<sep>",
  "sp_model_kwargs": {},
  "tokenizer_class": "XLNetTokenizer",
  "unk_token": "<unk>"
}
```

... which led to an even stranger error:

```
>>> from transformers import XLNetTokenizer
>>> tokenizer = XLNetTokenizer.from_pretrained("tokenizer", local_files_only=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hiekense/.local/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1811, in from_pretrained
    return cls._from_pretrained(
  File "/home/hiekense/.local/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1965, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/hiekense/.local/lib/python3.9/site-packages/transformers/models/xlnet/tokenization_xlnet.py", line 179, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/hiekense/.local/lib/python3.9/site-packages/sentencepiece/__init__.py", line 905, in Load
    return self.LoadFromFile(model_file)
  File "/home/hiekense/.local/lib/python3.9/site-packages/sentencepiece/__init__.py", line 310, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string
```

Also, there's nothing wrong with the tokenizer itself - testing it with GPT2's tokenizer (`GPT2Tokenizer.from_pretrained("tokenizer", local_files_only=True)`) yielded no errors.

Thank you.

### Expected behavior

N/A
UPDATE: According to [this summary](https://huggingface.co/docs/transformers/tokenizer_summary#sentencepiece), XLNet uses SentencePiece tokenization; so, I tried swapping in a `SentencePieceBPETokenizer` instead of a `ByteLevelBPETokenizer` (you should really include `SentencePieceBPETokenizer` in the docs by the way... the only mention I could find of it was [here](https://discuss.huggingface.co/t/training-sentencepiece-from-scratch/3477/2)). I'm receiving the exact same issue though.

Also, looks like I didn't include the code for training the tokenizer above, so I'll drop it down here:

```
def batch_iterator(dataset):
        for i in dataset:
                yield i["text"]

def getTokenizer(train=True, train_dataset=None):
        tokenizer = None

        if train:
                tokenizer = SentencePieceBPETokenizer()

                print("Training tokenizer...")
                tokenizer.train_from_iterator(batch_iterator(train_dataset), show_progress=True, vocab_size=VOCAB_SIZE, min_frequency=2, special_tokens=[
                        "<s>",
                        "<pad>",
                        "</s>",
                        "<unk>",
                        "<mask>",
                        NEWLINE
                ])
                print("Training complete. Saving tokenizer...")

                tokenizer.save_model("tokenizer")
```

... and my dataset:

```
Dataset({
    features: ['text'],
    num_rows: 2080000
})
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.