T5TokenizerFast not reversible when text contains special tokens
## Environment info

- `transformers` version: 4.8.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.8
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Who can help
@patrickvonplaten, @patil-suraj


## Information

I am using `T5TokenizerFast` initialized with `t5-base` tokenizer.

## To reproduce

```
from transformers import T5TokenizerFast


def main():
    tokenizer = T5TokenizerFast.from_pretrained('t5-base')

    """Note that all those strings will be decoded to the same string!"""
    s1 = "Hello <unk>world"
    s2 = "Hello<unk> world"
    s3 = "Hello <unk> world"
    s4 = "Hello<unk>world"
    for s in [s1, s2, s3, s4]:
        assert tokenizer.decode(tokenizer(s)['input_ids']) == 'Hello<unk> world</s>'


if __name__ == "__main__":
    main()
```
Heey @zorikg,

Is this a problem in your case? Some special tokens always strip away the space on the left so that we can assure the same expected behavior for the two use cases. *e.g.* when thinking about "<mask>" prediction, some users process the text in the form:

"`The capital of <mask> is Paris`" while others use "`The capital of<mask> is Paris`"

=> we want both cases to yield the correct <mask> token = France so that for some special tokens we think it's better to just always strip away the white space on the left (could be on the right as well)

<|||||>Hey @patrickvonplaten,

In your example both cases contain the same string "The capital of is Paris" (typo?) and I am not sure what  is the difference between them, could you clarify?

It seems that you don't only strip away white space form the left, you also add a white space to the right.

This is indeed a problem in my use case. I work under the assumption that sentence piece tokenizer should be fully reversible, which meant that `detokenize(tokenize(x)) == x`. 

In my scenario I look for answer spans inside a paragraph and checking if it contains certain subtext. I have many bugs around this issue when text contains unknown tokens. For example:
```
tokenizer = T5TokenizerFast.from_pretrained('t5-base')
s = 'maternal grandfather ʻAikanaka'
s_encode_decode = tokenizer.decode(tokenizer(s)['input_ids'])
print(s_encode_decode)
s_encode_decode_2 = tokenizer.decode(tokenizer(s_encode_decode)['input_ids'])
print(s_encode_decode_2)
```
The first print is `maternal grandfather <unk>Aikanaka</s>` and the second is `maternal grandfather<unk> Aikanaka</s></s>`. 

Due to the fact that I may encode & decode the paragraph and the answer many times I had many cases where  I thought that string did not contain certain substring but it actually did (because spaces around the <unk> token were mismatched). 

I do think that the contract should be that if I encode and decode I get the same result. If there are other use cases, I would consider supporting them with explicit argument, right now I feel that the API is a bit misleading and it actually took us a lot of time to figure out the reason for our bug. WDYT?

Thanks!<|||||>Sorry @zorikg, I forgot to put the text in `code format`. Now my example above should make more sense...

But looking more into it, I think this looks like a bug to me...

The following should work IMO:

```python
from transformers import T5TokenizerFast, AddedToken

tokenizer = T5TokenizerFast.from_pretrained('t5-base')
tokenizer.unk_token = AddedToken("<unk>", lstrip=False, rstrip=False)

s = "Hello <unk>there"

tokenizer.decode(tokenizer(s)['input_ids']) == s
```

cc'ing @LysandreJik @SaulLu - what do you think about this?<|||||>Thanks for the answer @patrickvonplaten. 

Unfortunately I ran your code and `tokenizer.decode(tokenizer(s)['input_ids']) == s` returns `False` :(

Also it seems that the type of `tokenizer.unk_token` is `str` and not `AddedToken`.

Do you have other workaround? Also I need this behavior to be consistent for all special tokens, including the `eos` token and all the additional tokens with special ids.<|||||>Thank you for the issue @zorikg , 

Indeed, I share your point of view, this behaviour is surprising. 

As a side note, because it could cause you some problems, tokenizers have not been designed with the idea of being reversible (the normalization operation can be non-reversible).

After some research, I don't see a solution to achieve what you want. Maybe @n1t0  pr @Narsil has an idea on the `tokenizer_backend` side ?

I think we should spend some time investigating how the `rstrip` and `lstrip` attributes are taken into account as the output does not seem natural to me. For example, on this example, I don't understand why 1) there is a not `"▁"` between `'▁grandfather'` and `'<unk>'` and 2) the `"▁A"` start with a `"▁"`. 
```python
tokenizer = T5TokenizerFast.from_pretrained('t5-base', unk_token=AddedToken("<unk>", lstrip=False, rstrip=False))
s = 'maternal grandfather <unk>Aikanaka'

s_encode_decode_tokens = tokenizer.convert_ids_to_tokens(tokenizer(s)['input_ids'])
```
Output:
```
['▁maternal', '▁grandfather', '<unk>', '▁A', 'i', 'kan', 'aka', '</s>']
```
[Edit]: as it is written in the documentation: 
> lstrip (bool, defaults to False) – Defines whether this token should strip all potential whitespaces on its left side. If True, this token will greedily match any whitespace on its left. For example if we try to match the token [MASK] with lstrip=True, in the text "I saw a [MASK]", we would match on " [MASK]". (Note the space on the left).

> rstrip (bool, defaults to False) – Defines whether this token should strip all potential whitespaces on its right side. If True, this token will greedily match any whitespace on its right. It works just like lstrip but on the right.
<|||||>Hmm, looked into it a bit, it's not `transformers` that's swallowing the extra space, it's T5 specific.

To do that, I checked with the slow tokenizer to get `tokenizers` out of the equation.
If you look at that, within `src/transformers/tokenization_utils_base.py::tokenize` you can check that everything gets split properly, BUT the t5 tokenization uses for `_tokenize` : `self.sp_model.encode(text, out_type=str)`

And if you check, 
```python
# notice extra space
self._tokenize("maternal grandfather ", out_type=str)
# ['▁maternal', '▁grandfather']
# SPACE GONE
```

The fact that `tokenizers` just replicates that behavior seems ok to me.
Anyway, the culprit is NOT transformers/tokenizers but really `T5`. (or `spm`)
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.