Bug in T5 Tokenizer - Adding space after special tokens
## Environment info

- `transformers` version: 4.16.1
- Platform: Linux-4.15.0-65-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.2+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help
@SaulLu @patrickvonplaten @PhilipMay 


## Information

When decoding with the T5 tokenizer it inserts redundant spaces after special tokens; this leads to incorrect decoding. Here is an example:

```bash
>> tokenizer.convert_ids_to_tokens(self.encode("``America''"))
['▁', '<unk>', 'America', "'", "'", '</s>']

>>self.convert_ids_to_tokens(self.encode(self.decode(self.encode("``America''"))))
['<unk>', '▁America', "'", "'", '</s>']
```

This happens in line 291 of file `tokenization_t5.py`: 
```python
out_string += self.sp_model.decode_pieces(current_sub_tokens) + token + " "
```

What is the purpose of the last space (" ")?

## Expected behavior

IMO the decoding in this example should be exactly the same as the original text. 

Thank you! :) 

That's a good question! @SaulLu @LysandreJik @n1t0 - do you have an idea by any chance?<|||||>This seems to be about special tokens:

```python
            # make sure that special tokens are not decoded using sentencepiece model
            if token in self.all_special_tokens:
                out_string += self.sp_model.decode_pieces(current_sub_tokens) + token + " "
                current_sub_tokens = []
```

https://github.com/huggingface/transformers/blob/2ca6268394c1464eaa813ccb398c3812904e2283/src/transformers/models/t5/tokenization_t5.py#L289-L292<|||||>@PhilipMay will removing the additional `" "` lead to any negative effect?

If not, can we please remove the additional `" "`? 
Sometimes we need to decode encoded inputs, and we will prefer the outcome to be as similar as possible to the original text.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.