LlamaTokenizerFast report vocab_size info not correct 
### System Info

Linux 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

    model = AutoModelForCausalLM.from_pretrained(
            base_model,
            load_in_8bit=True,
            device_map=device_map,
            trust_remote_code=True
        )
    # Tokenizer
    tokenizer = AutoTokenizer.from_pretrained(
        base_model,
        padding_side="right"

    )
return  ,LlamaTokenizerFast token 
    print(f" token class ,tokens vocab :{len(tokenizer.get_vocab())}\n {tokenizer}")
    

**token class ,tokens vocab :32001**
 LlamaTokenizerFast(name_or_path='/models/WizardLM-13B-V1.0-Merged', **vocab_size=32000**,(mistake ) model_max_length=2048, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'eos_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'unk_token': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'pad_token': '[PAD]'}, clean_up_tokenization_spaces=False)


### Expected behavior

with Vicuna or WizardLM-13B ,LlamaTokenizerFast report  vocab_size=32000 ,but should 32001,because PAD token added by   Vicuna


LlamaTokenizerFast(name_or_path='/models/WizardLM-13B-V1.0-Merged', **vocab_size=32000,** model_max_length=2048, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'eos_token': AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'unk_token': AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True), 'pad_token': '[PAD]'}, clean_up_tokenization_spaces=False)

cc @ArthurZucker <|||||>Hey! Sorry, the formatting of the issue is a bit strange I do not understand the problem. 
Could you try to reformulate the issue with a minimal reproducing script using a model path? <|||||>OK ,  base_model=/models/WizardLM-13B-V1.0-Merged 

    model = AutoModelForCausalLM.from_pretrained(
            base_model,
            load_in_8bit=True,
            device_map=device_map,
            trust_remote_code=True
        )
    # Tokenizer
    tokenizer = AutoTokenizer.from_pretrained(
        base_model,
        padding_side="right"
    )
   print(f" token class ,tokens vocab :{len(tokenizer.get_vocab())}\n {tokenizer}")
    print(tokenizer.get_vocab()）
 print( model ）
　ｏｕｔ　ｐｕｔ　

**token class ,tokens vocab :32001**
 LlamaTokenizerFast(name_or_path='/models/WizardLM-13B-V1.0-Merged', **vocab_size=32000,**

origin model
 LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(**32001**, 5120, padding_idx=0)

<|||||>So, 1. you still have not provided a link to a valid model on the hub. `WizardLM-13B-V1.0-Merged` does not seem to exist as it is probably a local folder on your machine. 2. if it is on the hub, feel free to open an issue on the corresponding repository, as this does not seem to be an issue with transformers. 
The llama2 models have the correct dimension. <|||||>https://huggingface.co/WizardLM/WizardLM-13B-V1.0   This is WizardLM-13B V1.0 diff weight. ,can using   
Alpaca also can used test
The Stanford Alpaca model independently trained on decapoda-research/llama-7b-hf at "chavinlo/alpaca-native" uses tokenizer.add_special_tokens({'pad_token': '[PAD]'}) and hence the model's vocab size is set to 32001.<|||||>I am really sorry but I don't understand the problem <|||||>with https://huggingface.co/WizardLM/WizardLM-13B-V1.0  or Alpaca model，vocab size is 32001，
but LlamaTokenizerFast report vocab_size=32000 (print(tokenlizer) out put result)
token class ,tokens vocab :32001
LlamaTokenizerFast(name_or_path='/models/WizardLM-13B-V1.0-Merged', vocab_size=32000,<|||||>I just ran:
```python 
>>> from transformers import LlamaTokenizerFast
>>> tok = LlamaTokenizerFast.from_pretrained("WizardLM/WizardLM-13B-V1.0")

>>> len(tok)
32001
```
Even if you want to force the vocab_size using : 
```python 
>>> tok = LlamaTokenizerFast.from_pretrained("WizardLM/WizardLM-13B-V1.0", vocab_size=32000)
```
this will not work. The fast tokenizer is initialized from the json file, and the `vocab_size` is not an argument that is used. If you don't want the padding token just set it to None and remove it from the `added_tokens_encoder` and `added_tokens_decoder`. 
