Decoding adds space between special tokens when skip_special_tokens = True
### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.15.109+-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (True)
- Tensorflow version (GPU?): 2.11.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.8
- JaxLib version: 0.4.7
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: -

### Who can help?

 @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "lmsys/fastchat-t5-3b-v1.0"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast = False)

inp = tokenizer.encode("Hello world")
print(inp)

out = tokenizer.decode(inp, skip_special_tokens = False,spaces_between_special_tokens=True)
print(out)
> 'Hello   world</s>'

out = tokenizer.decode(inp, skip_special_tokens = True,spaces_between_special_tokens=True)
print(out)
> 'Hello   world'

out = tokenizer.decode(inp, skip_special_tokens = True,spaces_between_special_tokens=False)
print(out)
> 'Hello world'


```

### Expected behavior

I expected the last two outputs to be same. In the 2nd last output, Since special tokens are skipped, no space should have been added even if `spaces_between_special_tokens=True`
Hey! The behaviour is totally expected: you added `" "` as an added token. This means that it will not be split, and will not be encoded directly using the underlying model. 
Here is what happens: (check [this](https://github.com/ArthurZucker/transformers/blob/74cb2b1dd9755f6bd72fc9e65d1e374e287afd84/src/transformers/models/t5/tokenization_t5.py#L324-L327)

```python 
>>> tokenizer.encode("Hello world")
# 1. Split the input using a trie made of special tokens and added tokens: 
["Hello", " ", "world"]
# 2. Add a prefix space
[" Hello", " ", " world"]
# 3. Replace prefix space with meta-space
['▁Hello', ' ', '▁world']
# 4. Get the corresponding tokens
[8774, 32106, 296, 1]
>>> tokenizer.decode(inp, skip_special_tokens = False,spaces_between_special_tokens=True)
# When decoding, `convert_tokens_to_string` is called. A ` ` is added before every special token. But ` ` is a special token, so a space will be added to it
["▁Hello", "  ", "▁world"]
```
When you skip special tokens, `" "` is skiped, but you still have the space from the tokenizer that *joins*  the text on a space to properly reformat it.

You are not correctly using the API, I would recommend you to remove "  " from the special tokens. 