Difference in decoded strings between a tokenizer and the corresponding fast tokenizer
## Environment info

- `transformers` version: 4.2.0
- Platform: Linux-4.15.0-130-generic-x86_64-with-debian-10.5
- Python version: 3.7.8
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 tokenizers: @mfuntowicz

## Information

I want to feed a word-based sequence to a tokenizer and get a word-based output decoded from logits.
To leave spaces before punctuation marks, I specified `tokenizer.decode(ids, clean_up_tokenization_spaces=False)`, but a fast tokenizer removes such spaces while the corresponding non-fast tokenizer preserves them.

## To reproduce

```py
from transformers import BertTokenizer, BertTokenizerFast

seq = ['Cheerfully', ',', 'Hello', 'World', '!']

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
ids = tokenizer(seq, is_split_into_words=True).input_ids
print(ids)  # => [101, 20394, 8284, 5834, 117, 8667, 1291, 106, 102]
print(tokenizer.decode(ids, clean_up_tokenization_spaces=False))  # => [CLS] Cheerfully , Hello World ! [SEP]

tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
ids = tokenizer(seq, is_split_into_words=True).input_ids
print(ids)  # => [101, 20394, 8284, 5834, 117, 8667, 1291, 106, 102]
print(tokenizer.decode(ids, clean_up_tokenization_spaces=False))  # => [CLS] Cheerfully, Hello World! [SEP]
```

This happens because the underlying tokenizer ([huggingface/tokenizers](https://github.com/huggingface/tokenizers/)) removes them at the [transformers/tokenization_utils_fast.py#L495](https://github.com/huggingface/transformers/blob/v4.2.0/src/transformers/tokenization_utils_fast.py#L495), whether `clean_up_tokenization_spaces` is `True` or `False`.

To avoid this issue, I tried to use `tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))`, but this also did not work.

## Expected behavior

A tokenizer and its corresponding fast tokenizer must return the same decoded string.

For the `WordPiece` decoder, which is used in `BertTokenizerFast`, It seems that `cleanup` cannot be changed after initialization.
https://github.com/huggingface/tokenizers/blob/python-v0.10.0/tokenizers/src/tokenizer/mod.rs#L762
https://github.com/huggingface/tokenizers/blob/python-v0.10.0/tokenizers/src/decoders/wordpiece.rs#L35
https://github.com/huggingface/tokenizers/blob/python-v0.10.0/bindings/python/py_src/tokenizers/decoders/__init__.pyi#L113
https://github.com/huggingface/transformers/blob/v4.2.0/src/transformers/convert_slow_tokenizer.py#L106

I confirmed that a tokenizer and the fast tokenizer return the same string when they are based on SentencePiece because it treats whitespace as a symbol and can reconstruct the original sentence.
So when specifying `clean_up_tokenization_spaces=False`, spaces before punctuation depend on `ids`, but there are no differences in the decoded string between a tokenizer (e.g. `T5Tokenizer`) and the fast tokenizer (e.g. `T5TokenizerFast`).<|||||>This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.