Incosistent vocab sizes in t5-base model & tokenizer
## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.0+cu111 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

I'm using `t5-base` model (yet my test shows the same result for `t5-small`).

To implement some knowledge-distillation task and "classical" sequence-to-sequence tasks - I'm trying to use `batch_size x seq_length x vocab_size` array as labels for both kinds of tasks (soft labels in KD case, one-hot hard labels I'm seq2seq case).

So I need to convert tokenizer output from `batch_size x seq_length` to `batch_size x seq_length x vocab_size` one-hot array to pass it to my custom loss later.
Yet I found out that I can't just use `T5Tokenizer(...).vocab_size` to build one-hot matrix - `T5ForConditionalGeneration(...).config.vocab_size` have different value.
So when I'm trying to build a one-hot vector based on tokenizer vocab size - I'm getting dimension mismatch errors.

The next code shows me different vocabulary sizes when I'm trying to access vocab size through `T5Tokenizer` and related `T5ForConditionalGeneration` config:
```
from transformers import T5ForConditionalGeneration, T5Tokenizer

T5_MODEL = "t5-base"
print("tokenizer", T5Tokenizer.from_pretrained(T5_MODEL).vocab_size)
print("model", T5ForConditionalGeneration.from_pretrained(T5_MODEL).config.vocab_size)
```
```
tokenizer 32100
model 32128
```

## Expected behavior

I expected the tokenizer & model to have the same vocabulary size.
cc @patrickvonplaten @patil-suraj <|||||>Duplicate of https://github.com/huggingface/transformers/issues/4875 I think<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.