T5 Tokenizer Prepends Space after Each Added (Extra) Token
### System Info

```
$ transformers-cli env

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.22.1
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.31
- Python version: 3.10.7
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.0+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@LysandreJik @SaulLu 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('t5-base')
tokenizer.add_tokens(['<']) # '>' is already in the vocab

tokenizer.decode(tokenizer('a>=5').input_ids)
# prints 'a>=5</s>' as expected (no space after >)
tokenizer.decode(tokenizer('a<=5').input_ids)
# prints 'a< =5</s>'

### Expected behavior

There shouldn't be a space after the `<` character.
In case it helps with debugging, it returns a different result with `use_fast=False`:

```
tokenizer = AutoTokenizer.from_pretrained('t5-base', use_fast=False)
tokenizer.add_tokens(['<'])
tokenizer.decode(tokenizer('a<=5').input_ids)
# 'a < =5</s>' (notice the space both before and after the `<`)<|||||>Maybe also of interest to @ArthurZucker <|||||>I did some digging, and I believe the culprit (in the non-fast, but potentially fast, tokenizer) is this:
https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils.py#L433

```
# Make sure we don't split on any special tokens (even they were already in the vocab before e.g. for Albert)
if special_tokens:
    if len(new_tokens) == 1:
        _insert_one_token_to_ordered_list(self.unique_no_split_tokens, new_tokens[0])
    else:
        self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(new_tokens)))
else:
    # Or on the newly added tokens
    if len(tokens_to_add) == 1:
        _insert_one_token_to_ordered_list(self.unique_no_split_tokens, tokens_to_add[0])
    else:
        self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(tokens_to_add)))
self._create_trie(self.unique_no_split_tokens)
```

Do you know if that's a fundamental restriction with respect to the models? Or if not, could we potentially expose a flag that disables this behavior?<|||||>@ankrgyl  What is the expected behaviour of decode should be ? <|||||>My desired behavior would be that `a<=5` round trips (i.e. encodes and then decodes) to `a<=5`, not `a < =5`<|||||>@LysandreJik  @ArthurZucker I debugged this issue. 

Incase of of input is a a>=5
calling self.tokens_trie.split() returns ['a>=5']

Incase of of input is a a<=5
calling self.tokens_trie.split() returns ['a', '<', '=5']

Here '<' is not part of the original vocab is added. <|||||>Hi @raghavanone just to clarify, that has been known the whole time. The specific issue here is that added tokens (in this case `<`) _cannot_ split words (see the code snippet I pasted above).<|||||>@ankrgyl Did some more digging, this is being done intentionally when we add new token to tokenizer. Look at _decode function in tokenization_utils.py . I do not think this a bug. 

@ArthurZucker Please validate this understanding.<|||||>Hey! @raghavanone you are right, this is not a bug! 
@ankrgyl you can use the following snippet : 
```python 
>>> tokenizer.decode(tokenizer('a<=5').input_ids, spaces_between_special_tokens = False)
>>> 'a<=5</s>'
``` 
(this works for the slow tokenizer, not for the fast, will have a look) tell me if that fixes your issue! 😄 <|||||>@ArthurZucker  Just pondering why is slow and fast tokenizer functionally not equal ?<|||||>Well.. this is not really intended ^^ But mostly the `fast` is an entire library mostly implemented in `rust`, so we must have forgotten to update this argument when adding it to the `transformers` tokenizers. cc @LysandreJik  and @SaulLu FYI 🤗 <|||||>@ArthurZucker confirmed with both the slow and fast tokenizers (`build_tokenizer()` below is a wrapper function in my code that simply adds `<` as a special token):

```
In [2]: tokenizer = model.build_tokenizer()

In [3]: tokenizer.decode(tokenizer('a<=5').input_ids, spaces_between_special_tokens = False)
Out[3]: 'a< =5</s>'

In [4]: tokenizer = model.build_tokenizer(use_fast=False)

In [5]: tokenizer.decode(tokenizer('a<=5').input_ids, spaces_between_special_tokens = False)
Out[5]: 'a<=5</s>'
```<|||||>Awesome, closing this issue, will open a PR in tokenizers when I have bandwith to try to match the outputs. <|||||>@ArthurZucker happy to help with this