fsmt Tokenizer.save_vocabulary Bug
### System Info

- `transformers` version: 4.26.1
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm using FSMT and try to reproduce the behavior of basic transformer on WMT16 translation task with [run_translation.py](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation). It turns out that during the training, both valid loss and training loss go down normally, and valid BLEU scores increase gradually at the beginning, but suddenly dropped a lot and corrupt to ~0. After some investigation, I found the fsmt_tokenizer keeps complaining each time I try to save the training states. 

You should be able to reproduce the complaint as easy as running the short script below: 

```
from transformers import (
    AutoModelForSeq2SeqLM,
    AutoTokenizer
    )

tokenizer = AutoTokenizer.from_pretrained("allenai/wmt16-en-de-12-1")

tokenizer.save_vocabulary("./tmp")
```

### Expected behavior

```
Saving vocabulary to ./tmp/merges.txt: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!
```
And the training curves valid BLEU looks like [this](https://api.wandb.ai/links/alanlee/2zddajjr) and valid loss looks like [this](https://api.wandb.ai/links/alanlee/h77vb3bf), which show that there should be no gradient explosion but still broken.
Hey, apparently, it is a known issue that the vocabulary has some holes. The problem with your training is probably not related as you mention that it `suddendly` drops and corrupts. Meaning up until some point everything works well no? 😉 <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.