Fast tokenizer converter leads to PanicException: no entry found for key
I am working on adding PLBart's tokenizer. The tokenizer uses `sentencepiece.bpe.model` and is similar to MBart. Hence, to convert to fast tokenizer, I used the same converter - `MBartConverter` and modified it. The definition is as follows and can also be found [here](https://github.com/huggingface/transformers/blob/5c309845fa943e228e3afa2ca4b781e3b878ad8f/src/transformers/convert_slow_tokenizer.py#L896-L920):
```python
class PLBartConverter(SpmConverter):
    def vocab(self, proto):
        vocab = [
            ("<s>", 0.0),
            ("<pad>", 0.0),
            ("</s>", 0.0),
            ("<unk>", 0.0),
        ]
        vocab += [(piece.piece, piece.score) for piece in proto.pieces[3:]]
        vocab += [("java", 0.0), ("python", 0.0), ("en_XX", 0.0)]
        vocab += [("<mask>", 0.0)]
        return vocab

    def unk_id(self, proto):
        return 3

    def post_processor(self):
        return processors.TemplateProcessing(
            single="$A </s> en_XX",
            pair="$A $B </s> en_XX",
            special_tokens=[
                ("en_XX", self.original_tokenizer.convert_tokens_to_ids("en_XX")),
                ("</s>", self.original_tokenizer.convert_tokens_to_ids("</s>")),
            ],
        )
```

However, running the conversion method 
```python
from transformers.convert_slow_tokenizers_checkpoints_to_fast import convert_slow_checkpoint_to_fast
convert_slow_checkpoint_to_fast('PLBartTokenizer','plbart-base', 'plbart-base', False)
```
leads to the following error:

```python
Assigning ['java', 'python', 'en_XX'] to the additional_special_tokens key of the tokenizer
Save fast tokenizer to plbart-base with prefix plbart-base add_prefix True
=> plbart-base with prefix plbart-base, add_prefix True
tokenizer config file saved in plbart-base/plbart-base-tokenizer_config.json
Special tokens file saved in plbart-base/plbart-base-special_tokens_map.json
thread '<unnamed>' panicked at 'no entry found for key', /__w/tokenizers/tokenizers/tokenizers/src/models/mod.rs:36:66
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/crocoder/Desktop/transformers/src/transformers/convert_slow_tokenizers_checkpoints_to_fast.py", line 87, in convert_slow_checkpoint_to_fast
    file_names = tokenizer.save_pretrained(
  File "/home/crocoder/Desktop/transformers/src/transformers/tokenization_utils_base.py", line 2044, in save_pretrained
    save_files = self._save_pretrained(
  File "/home/crocoder/Desktop/transformers/src/transformers/tokenization_utils_fast.py", line 579, in _save_pretrained
    self.backend_tokenizer.save(tokenizer_file)
pyo3_runtime.PanicException: no entry found for key
```

### Possible Fixes?
- https://github.com/huggingface/tokenizers/issues/776 - Suggests removing `special_tokens` from the trainer. 
  I assumed  that is analogous  to removing `special_tokens` from the `post_processor`? I tried it and it leads to the following error:
  ```python
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/home/crocoder/Desktop/transformers/src/transformers/convert_slow_tokenizers_checkpoints_to_fast.py", line 60, in convert_slow_checkpoint_to_fast
      tokenizer = tokenizer_class.from_pretrained(checkpoint, force_download=force_download)
    File "/home/crocoder/Desktop/transformers/src/transformers/tokenization_utils_base.py", line 1744, in from_pretrained
      return cls._from_pretrained(
    File "/home/crocoder/Desktop/transformers/src/transformers/tokenization_utils_base.py", line 1872, in _from_pretrained
      tokenizer = cls(*init_inputs, **init_kwargs)
    File "/home/crocoder/Desktop/transformers/src/transformers/models/plbart/tokenization_plbart_fast.py", line 138, in __init__
      super().__init__(
    File "/home/crocoder/Desktop/transformers/src/transformers/models/xlm_roberta/tokenization_xlm_roberta_fast.py", line 134, in __init__
      super().__init__(
    File "/home/crocoder/Desktop/transformers/src/transformers/tokenization_utils_fast.py", line 111, in __init__
      fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
    File "/home/crocoder/Desktop/transformers/src/transformers/convert_slow_tokenizer.py", line 1056, in convert_slow_tokenizer
      return converter_class(transformer_tokenizer).converted()
    File "/home/crocoder/Desktop/transformers/src/transformers/convert_slow_tokenizer.py", line 488, in converted
      post_processor = self.post_processor()
    File "/home/crocoder/Desktop/transformers/src/transformers/convert_slow_tokenizer.py", line 913, in post_processor
      return processors.TemplateProcessing(
  ValueError: Missing SpecialToken(s) with id(s) `</s>, en_XX`
  ```

### Related issues
- https://github.com/huggingface/tokenizers/issues/611
- While this error can be found in #13443, I am unable to understand how to fix an existing `sentencepiece.bpe.model` file to remove non-consecutive tokens, if that is the case.
- https://github.com/huggingface/tokenizers/issues/260 - Similar suggestion, but unclear what to do in case of a spm file.


Hey @gchhablani,

Thanks a lot for the in-detail issue! Could we in a first step upload the slow tokenizer to verify that it works correctly? In a next step we can then convert it to a fast tokenizer <|||||>Given that `mbart`  has a  very specific tokenization, we might have to add a new `tokenization_plbart.py` file in my opinion. Or do you think the MBart tokenizer is 1-to-1 correct for PLBart's tokenization?<|||||>The folder `plbart` could then look similar to this: https://github.com/huggingface/transformers/tree/master/src/transformers/models/barthez

<|||||>Also cc'ing @patil-suraj here since it's very much MBart related<|||||>Hey Gunjan! Could you share the PR so I could take a  look at the tokenizer code?<|||||>@patrickvonplaten I think PLBart tokenizer is similar to MBart tokenizer. There are two kinds of tokenizers needed, the modeling file is also different from MBart in that it uses MBart style model (`shift_tokens_right`) but follows Bart style architecture. I have create two models `PLBart` and `PLBartMulti` which are present in the branch as of now.

@patil-suraj The PR can be found [here](https://github.com/huggingface/transformers/pull/13269).<|||||>Unstale<|||||>Taking care of it on Monday next week<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>@patil-suraj - could you take this over since you're working on the PLBart integration?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.