T5Tokenizer: decode does not show special tokens
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: N/a
- Using distributed or parallel set-up in script? N/a

### Who can help
  examples/seq2seq: @sshleifer
 -->

## Information

Model I am using (Bert, XLNet ...): T5Tokenizer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```python
from transformers import T5Tokenizer
input = "word  <pad> <unk> </s> </s>"
t5tokenizer = T5Tokenizer.from_pretrained('t5-small')
tokenized = t5tokenizer.batch_encode_plus([input], max_length=10, padding="longest", return_tensors="pt").input_ids
print(t5tokenizer.batch_decode(tokenized, skip_special_tokens=False, clean_up_tokenization_spaces=False))
```
IDs output: ` _word  <pad> <unk> </s> </s>`
decode output: `word  ⁇ `


## Expected behavior
The tokens should be shown in the decoded output, but everything except for the unknown token is dropped (no pad or EOS). 
`convert_ids_to_tokens` followed by `convert_tokens_to_string` also drops the tokens.
<!-- A clear and concise description of what you would expect to happen. -->

T5: @patrickvonplaten I think you need to set `_additional_special_tokens`.<|||||>@jsrozner want to try to fix?<|||||>This is a duplicate of #5142 and will be fixed with the PR linked below. Thanks for reporting it - seems like multiple people were running into this issue!!!