[BUG] RecursionError: maximum recursion depth exceeded when loading LLaMA-1 but success loading LLaMA-2
### System Info

- `transformers` version: 4.31.0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Running (Open-Assistant)[https://github.com/LAION-AI/Open-Assistant] script for (SFT)[https://github.com/LAION-AI/Open-Assistant/blob/main/model/model_training/trainer_sft.py]
When loading LLaMA-1, got this error:
```
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1155, in unk_token_id
    return self.convert_tokens_to_ids(self.unk_token)
RecursionError: maximum recursion depth exceeded
```
Seems like go into a infinite loop, and success loading LLaMA-2.

### Expected behavior

Success loading LLaMA-1 and LLaMA-2
I set a small recursion limit to 50 and I now got this trace back for loading LLaMA-1:
```
  File "/mnt/home//Open-Assistant/model/model_training/trainer_sft.py", line 481, in <module>
    main()
  File "/mnt/home//Open-Assistant/model/model_training/trainer_sft.py", line 333, in main
    tokenizer = get_tokenizer(training_conf)
  File "/mnt/home//Open-Assistant/model/model_training/utils/utils.py", line 214, in get_tokenizer
    tokenizer = transformers.AutoTokenizer.from_pretrained(tokenizer_name, cache_dir=conf.cache_dir)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 702, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1841, in from_pretrained
    return cls._from_pretrained(
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2004, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 126, in __init__
    self.update_post_processor()
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/models/llama/tokenization_llama_fast.py", line 136, in update_post_processor
    bos_token_id = self.bos_token_id
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1136, in bos_token_id
    return self.convert_tokens_to_ids(self.bos_token)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 250, in convert_tokens_to_ids
    return self._convert_token_to_id_with_added_voc(tokens)
  File "/mnt/data/conda/envs/oasst/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py", line 257, in _convert_token_to_id_with_added_voc
    return self.unk_token_id
...
```
Until it exceeded the limit.<|||||>FYI, I can load LLaMA-1 using `transformers==4.28.1`<|||||>Hey! Thanks for reporting, this is very similar to #22415, and #22414.
Having a minimal reproducer would be better, I cannot reproduce this out-of-the-box, have no idea what `tokenizer_config` they are using or what not. 

From the traceback, I created a minimal reproducer:
```python 
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", unk_token = "<endoftext>", bos_token="<endoftext>", eos_token="<endoftext>")
```
In this case, the `<endoftext>` token does not exist, and since there are a few issues with adding tokens when initializing, cf #23909 after calling `super().__init__()` the token is still not part of the vocab. 


It works with  `transformers==4.28.1`, because the tokenizer did not have the `self.update_post_processor()`. 

There are no real quick fixes appart from downgrading for now, but I'll probably either remove the call in the init, or fixing token addition will make sure the token is properly added after calling `super().__init__()`. <|||||>@ArthurZucker 
To reproduce:
`tokenizer_config.json`:
```json
{"bos_token": "", "eos_token": "", "model_max_length": 1000000000000000019884624838656, "tokenizer_class": "LlamaTokenizer", "unk_token": ""}
```
This is an old version of LLaMA-1, I think it is from [here](https://huggingface.co/decapoda-research/llama-7b-hf/tree/main)(need to edit `LLaMATokenizer` to `LlamaTokenizer`).