[LLAMA]: LLAMA tokenizer
### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.13.2
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
from transformers import LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained(..., add_eos_token=True)
input_text = "Hello, Huggingface"
tokens = tokenizer(input_text)
# token.input_ids = [0, ..., 0]
```

### Expected behavior

at commit id ``7ade6ef7d``

the ``eos_token_id`` and ``bos_token_id`` is 0 and 0, while those from LLAMA official repo released by META is 2 and 1. 

Why there is such kind of distinction?
cc @ArthurZucker <|||||>Hey! This is because: 
1. You are not using the correct version of transformers:
2. You are not using the correct checkpoints
Here is the snipper I used to get the expected output:
```python 
>>> from transformers import LlamaTokenizer
>>> tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b", add_eos_token= True)
>>> tokenizer.decode(tokenizer.encode("Hello", add_special_tokens = True))
'<s> Hello</s>'
```

<|||||>> Hey! This is because:
> 
> 1. You are not using the correct version of transformers:
> 2. You are not using the correct checkpoints
>    Here is the snipper I used to get the expected output:
> 
> ```python
> >>> from transformers import LlamaTokenizer
> >>> tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b", add_eos_token= True)
> >>> tokenizer.decode(tokenizer.encode("Hello", add_special_tokens = True))
> '<s> Hello</s>'
> ```

Which version of transformers do you use?<|||||>`main` but latest release also has these changes<|||||>Faced the same issue. It seems like a mismatch between transformers and llama chkt version.

It appears that in commit c0f99b4d2ec73090595914dde4c16da207e21d73, a major change has been made to llama tokenizer, so you either install an earlier version (commit 9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0 or before), or convert llama weight using the latest commit. <|||||>Or you can just use the tokenizer files from `huggyllama` and save them wherever you want <|||||>> Faced the same issue. It seems like a mismatch between transformers and llama chkt version.
> 
> It appears that in commit [c0f99b4](https://github.com/huggingface/transformers/commit/c0f99b4d2ec73090595914dde4c16da207e21d73), a major change has been made to llama tokenizer, so you either install an earlier version (commit [9eae4aa](https://github.com/huggingface/transformers/commit/9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0) or before), or convert llama weight using the latest commit.

Many thanks! Already fix it.<|||||>> Faced the same issue. It seems like a mismatch between transformers and llama chkt version.
> 
> It appears that in commit [c0f99b4](https://github.com/huggingface/transformers/commit/c0f99b4d2ec73090595914dde4c16da207e21d73), a major change has been made to llama tokenizer, so you either install an earlier version (commit [9eae4aa](https://github.com/huggingface/transformers/commit/9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0) or before), or convert llama weight using the latest commit.

You have just saved my life! Debugged for days for this weird problem.<|||||>> Or you can just use the tokenizer files from `huggyllama` and save them wherever you want

Hi, how can I convert a llama model trained with older transformers (commit [9eae4aa](https://github.com/huggingface/transformers/commit/9eae4aa57650c1dbe1becd4e0979f6ad1e572ac0) or before) to be compatible with the latest transformer code?