T5 tokenizer.vocab_size and config.vocab_size mismatch?
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 4.1.1
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1
- tokenizers: 0.9.4
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


## Information
Hi @patrickvonplaten 
I am trying to train a "t5-base" model and I directly use from_pretrained tokenizer, config and model. However, I found the vocabulary size given by the tokenizer and config is different (see to reproduce). Does this is expected? 

If I use the model `T5ForConditionalGeneration.from_pretrained('t5-base', config=config)` to do predictions, this will result in the last dimension of lm_logits is different from `tokenizer.vocab_size`.


## To reproduce

Steps to reproduce the behavior:

```
>>> from transformers import T5Tokenizer, T5Config
>>> tokenizer = T5Tokenizer.from_pretrained("t5-base")
>>> config = T5Config.from_pretrained("t5-base")
>>> print(tokenizer.vocab_size)
32100
>>> print(config.vocab_size)
32128
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
```
>>> print(tokenizer.vocab_size)
32128
>>> print(config.vocab_size)
32128
```
Duplicate of https://github.com/huggingface/transformers/issues/4875.<|||||>I see, I simply ignored this mismatch and seems nothing wrong with prediction.
Thank you!