Mismatch between config.vocab_size and len(tokenizer) in Flan-T5
### System Info

- `transformers` version: 4.28.1
- Platform: Linux-5.15.0-1023-azure-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer,AutoConfig
models = [
    "google/flan-t5-small",
    "google/flan-t5-base",
    "google/flan-t5-large",
    "google/flan-t5-xl",
    "google/flan-t5-xxl",
]
for model in models:
    config = AutoConfig.from_pretrained(model)
    tokenizer = AutoTokenizer.from_pretrained(model)
    print(f"{model}\n\tlen(tokenizer)={len(tokenizer)},tokenizer.vocab_size={tokenizer.vocab_size},config.vocab_size={config.vocab_size}")
```
![draft ipynb — LaMP  SSH: v100node  2023-05-08 13-41-06](https://user-images.githubusercontent.com/38466901/236742905-813afcaa-e2ae-46fe-a6dd-612c45f04ae4.png)


### Expected behavior

The two are matched.
See #4875.<|||||>Got it. Thanks! Voting for clarification.<|||||>I gather from the thread that it shouldn't be a problem. The size was increased for the ease of GPU usage. Is this creating any issues in inferencing or training the model? Just want to know for better understanding!<|||||>Ok, here is my usecase.
I usually calculate loss myself rather than pass `labels` into a model for automatic loss calculation.
For example, when calculating commonly used **Cross Entropy Loss**, i would have to figure out vocab size myself.
```python
loss = F.cross_entropy(
                        outputs.logits.view(-1,VOCAB_SIZE),
                        labels.view(-1),
                        label_smoothing=cfg.trainer.label_smoothing_factor
                        )
```
So i think in cases where `vocab_size` matters, the value from from `config.vocab_size` and `len(tokenzier)` should be consistent.<|||||>The pre-trained model which is provided by google had its vocab_size manually set to 32128 by them. Here's what I found in their github:
<img width="809" alt="Screenshot 2023-05-20 at 9 54 23 AM" src="https://github.com/huggingface/transformers/assets/118152679/86b3bbf1-8d72-4b9f-807b-3ab6dfb3a2ef">
Did you try setting VOCAB_SIZE manually?<|||||>Hi, thanks for the response! I just mean when i need to know the value of `vocab_size`, imo it should be consistent with `len(tokenzier)` and `config.vocab_size` in huggingface.