T5 Tokenizer fails to decode correctly and prints ⁇
T5 Tokenizer tokenizes this following sequence to 
```
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("t5-base")
>>> print(tokenizer.tokenize("My phone number is  1-${phone.number}"))
['▁My', '▁phone', '▁number', '▁is', '▁1-', '$', '{', 'phone', '.', 'num', 'ber', '}']
```
So far so good but when we decode the above sequence back, we get weird ⁇ symbols.


```
>>> print(tokenizer.decode(tokenizer.encode("My phone number is  1-${phone.number}")))
My phone number is 1-$ ⁇ phone.number ⁇ 
```

This along with the bug https://github.com/huggingface/transformers/issues/6150 shows that T5 Tokenizer
- Is not cycle consistent
- Ignores multiple whitespaces
##  Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.0.2                                                                                                                                                            
- Platform: Linux-4.9.0-12-amd64-x86_64-with-Ubuntu-18.04-bionic                                                                                                                           
- Python version: 3.6.9                                                                                                                                                                    
- PyTorch version (GPU?): 1.5.1+cpu (False)                                                                                                                                                
- Tensorflow version (GPU?): not installed (NA)                                                                                                                                            
- Using GPU in script?: no                                                                                                                                                     
- Using distributed or parallel set-up in script?: no 
### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 


 -->
T5: @patrickvonplaten
## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Shown above

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
A cycle consistent T5 Tokenizer that works on a variety of inputs 
@mfuntowicz - Since T5 relies on google's sentencepiece tokenizer for now, can we do anything against it before our own sentencepiece tokenizer is implemented? <|||||>Verified that this is a problem with the original T5 sentencepience tokenizer. Opened an issue with the Google's T5 repository. https://github.com/google-research/text-to-text-transfer-transformer/issues/390<|||||>Closing this issue , quoting from T5 github issue
> > { is OOV because we intentionally removed any pages with { or } from C4 to avoid pre-training on anything other than natural language. So, it gets encoded to ??. SentencePiece has a byte fallback feature but it was not available when we trained our sentencepiece model.