Discrepancy between PegasusTokenizer and PegasusTokenizerFast
### System Info

- `transformers` version: 4.22.0.dev0
- Platform: Linux-5.8.0-51-generic-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@patil-suraj

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import PegasusTokenizerFast,PegasusTokenizer
toker_fast = PegasusTokenizerFast.from_pretrained("/data/pretrained_model/pegasus_large/")
toker = PegasusTokenizer.from_pretrained("/data/pretrained_model/pegasus_large/")
print(toker_fast.decode(toker_fast.encode("huggingface/transfomer"),skip_special_tokens=False)) ## huggingface/transfomer</s>
print(toker.decode(toker.encode("huggingface/transfomer"),skip_special_tokens=False)) ## huggingface/transfomer
```

### Expected behavior

These two should be the same. I suppose this is the problem of `PegasusTokenizer`. Because EOS token is needed in Generation Task.
I think this may be the problem of `PegasusTokenizer.decode()`. It indeed add `<\s>` to the end of the sentence, but decoder fails to keep it even when `skip_special_tokens=False`
<img width="819" alt="draft ipynb — selfmem  SSH: wict3090  2022-11-06 10-19-50" src="https://user-images.githubusercontent.com/38466901/200150974-1bde2180-8c6a-4ea0-94f8-6e36a20fceca.png">
<|||||>I think this was recently fixed by #15775. At least on `google/pegasus-xsum` and the main branch of Transformers, I don't see any differences in the outputs.
Not sure if this is the model you were using since yours is local. Could you give us a repo ID on the Hub if the issue persists on your side?<|||||>after updating transformers to the latest version by
```shell
pip uninstall transformers
pip install transformers
```
on `google/pegasus-large` and `google/pegasus-xsum`, the problem still exists:
![draft ipynb — selfmem  SSH: 45a3159k71 zicp vip  2022-11-07 23-31-11](https://user-images.githubusercontent.com/38466901/200349559-03707957-2f9c-4a80-9f06-909d27b79585.png)
![draft ipynb — selfmem  SSH: 45a3159k71 zicp vip  2022-11-07 23-32-12](https://user-images.githubusercontent.com/38466901/200349758-2f9a86a2-0aa0-433d-aa14-a307c164d2d5.png)
<|||||>Sorry, I didn't notice this PR has not been merged yet.
This https://github.com/huggingface/transformers/pull/15775 indeed solves this problem. Thanks !
![draft ipynb — selfmem  SSH: 45a3159k71 zicp vip  2022-11-07 23-45-40](https://user-images.githubusercontent.com/38466901/200352923-40e6ef4e-d60e-4441-98ca-ecd9aad2c061.png)
