T5 Base length of Tokenizer not equal config vocab_size 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: Installed from git

## Issue
The `len(AutoTokenizer.from_pretrained("t5-base"))` is `32100` but the `T5ForConditionalGeneration.from_pretrained("t5-base").config.vocab_size`  is `32128`. Seems to be a similar issue to that of : https://github.com/huggingface/transformers/issues/2020


duplicate of https://github.com/huggingface/transformers/issues/4875 I think<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.