AssertionError: Non-consecutive added token '<pad>' found. Should have index 40002 but has index 40000 in saved vocabulary
torch: 1.6.0
transformers: 3.5.1
OS: centos 7
GPU: A100

I trained a sentencepiece bpe model. There is no problem if I load it with `XLMRobertaTokenizer`. But when I load with `XLMRobertaTokenizerFast`, it cost a long time in `transformers/convert_slow_tokenizer.py`. After save the tokenizer and load it `from_pretrained`, the error occurs:
```
/ProjectRoot/tp_origin/pyenv/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs)
   1758             for token, index in added_tok_encoder_sorted:
   1759                 assert index == len(tokenizer), (
-> 1760                     f"Non-consecutive added token '{token}' found. "
   1761                     f"Should have index {len(tokenizer)} but has index {index} in saved vocabulary."
   1762                 )

AssertionError: Non-consecutive added token '<pad>' found. Should have index 40002 but has index 40000 in saved vocabulary.
```
I find that a file `added_tokens.json` is created with content `{"<pad>": 40000, "<mask>": 40001}`. 

my tokenizer
```
PreTrainedTokenizerFast(name_or_path='/ProjectRoot/tp_origin/distillation/tmp/student_init_model3/0_Transformer', vocab_size=40000, model_max_len=514, is_fast=True, padding_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': '<mask>'})
```
I think the problem is that the saved tokenizer saves `len(tokenizer)` = 40002. So when I load it, the added tokens id starts from 40000, the error occurs.<|||||>Hey @thesby,

Did you add any special tokens to `XLMRobertaTokenizer` that weren't there previously? 
Could you copy/paste the code you used to train the tokenizer here as well? Thanks!<|||||>All special tokens were added by sentencepiece trainer. I have never add by myself.
```
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib64/ spm_train --input=main.txt --model_prefix=sentencepice3.bpe --vocab_size=40000 --character_coverage=0.9995 --model_type=bpe --max_sentencepiece_length=4 --num_threads=64 --split_digits=true --input_sentence_size=2000000 -shuffle_input_sentence=true
```<|||||>Maybe I should train with `unigram`, not `bpe`.<|||||>@thesby  have you solved this problem?<|||||>Hi,

I'm having a similar problem. 

```
from transformers import GPT2Tokenizer

class VisualCometTokenizer(GPT2Tokenizer):
    def __init__(self,
                 vocab_file,
                 merges_file,
                 errors='replace',
                 unk_token="<|endoftext|>",
                 bos_token="<|endoftext|>",
                 eos_token="<|endoftext|>",
                 begin_img="<|b_img|>",
                 end_img="<|e_img|>",
                 begin_event="<|b_ev|>",
                 end_event="<|e_ev|>",
                 begin_place="<|b_pl|>",
                 end_place="<|e_pl|>",
                 begin_inferences={'before': "<|before|>", 'intent': "<|intent|>", 'after': "<|after|>"},
                 end_inference="<|e_in|>",
                 **kwargs):
        super(VisualCometTokenizer, self).__init__(
            vocab_file,
            merges_file,
            errors=errors,
            bos_token=bos_token,
            eos_token=eos_token,
            unk_token=unk_token,
            **kwargs
        )

        self.begin_img = begin_img
        self.end_img = end_img
        self.begin_event = begin_event
        self.end_event = end_event
        self.begin_place = begin_place
        self.end_place = end_place
        self.begin_inferences = begin_inferences
        self.end_inference = end_inference
        self.det_tokens = ['<|det%d|>' % i for i in range(50)]
        self.add_special_tokens({
            "additional_special_tokens": [self.begin_img, self.end_img, self.begin_event, self.end_event,
                                          self.begin_place, self.end_place, self.end_inference]
                                         + list(self.begin_inferences.values()) + self.det_tokens
        })

tokenizer = VisualCometTokenizer.from_pretrained("gpt2")
tokenizer.save_pretrained("/content/test")

tokenizer =VisualCometTokenizer.from_pretrained("/content/test")

```

Will cause this:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-34-e955f380827e> in <module>()
     46 tokenizer.save_pretrained("/content/test")
     47 
---> 48 tokenizer =VisualCometTokenizer.from_pretrained("/content/test")

1 frames
/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs)
   1809             for token, index in added_tok_encoder_sorted:
   1810                 assert index == len(tokenizer), (
-> 1811                     f"Non-consecutive added token '{token}' found. "
   1812                     f"Should have index {len(tokenizer)} but has index {index} in saved vocabulary."
   1813                 )

AssertionError: Non-consecutive added token '<|b_img|>' found. Should have index 50317 but has index 50257 in saved vocabulary.


++

I moved the "add_special_tokens" outside of init and it loads fine. You have to add tokens outside of init and save and when you load the tokenizer again, it won't try to add the tokens twice.

Better fix would be to permit adding the same token to the tokenizer again or throw a warning in huggingface's tokenizer_utils.py<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.