Decoding with skip_special_tokens=True doesn't remove pad token
### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: -

### Who can help?

@younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

collab nb - https://colab.research.google.com/drive/1VH-ZVITqJ5k6umvMXvQc2Suxzf6KEXyX?usp=sharing

### Expected behavior

I expected the <pad> token to not be present in the output, just the processed text
cc @ArthurZucker <|||||>@ArthurZucker can confirm, but I suspect the pad token of the tokenizer is probably not defined correctly, in the notebook you have:

```python
print(tokenizer.pad_token,tokenizer.pad_token_id)
>>> [PAD] 32100
```

and in the generated text: 
```python
print(decoded_outputs)
>>> ['<pad> bhubaneswar</s>', '<pad> Hippos are large animals</s><pad>']
```

You probably need to update the tokenizer with the correct pad token and eos tokens, in your case respectively `<pad>` and `</s>`<|||||>As mentioned by Younes, the padding token is not correct, `tokenizer.additional_special_tokens` does not have the `<pad>`  token, and the `tokenizier.pad_token` is not set to `<pad>`. Thus the token is an `AddedToken` which is why you can see it in the `tokenizer.added_tokens_encoder` for example, but it is not a special token, this it is skipped. <|||||>Thanks for the prompt reply, cool So this is an issue with the model's tokenizer not having the right initial configuration? - and fixing it either by the [author on the model repo](https://huggingface.co/lmsys/fastchat-t5-3b-v1.0) or doing the following step by the user is the immediate remediation for this:
`tokenizer.add_special_tokens({'pad_token' : "<pad>"})`

I can confirm that this fixes the default behaviour