Infernal tokenizer loading trained 
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.dev0
- Platform: Ubuntu 18
- Python version: 3.7
- PyTorch version (GPU?): 1.7.1 (YES)
- Tensorflow version (GPU?):
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
@LysandreJik @patrickvonplaten  @patil-suraj @sgugger @n1t0


## Information

Model I am using (Bert, XLNet ...): DeBerta

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)



The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Use, for example, the OSCAR corpus in spanish, then use Tokenizers library to train your BPETokenizer (the one Deberta needs).
2. Try to load DebertaTokenizer from the .json generated by Tokenizers.

The code used for training the tokenizer was the following:

```{python}
import glob
import os
import random

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer

if __name__ == "__main__":
    tokenizer = Tokenizer(BPE())
    # tokenizer = ByteLevelBPETokenizer(add_prefix_space=False)

    trainer = BpeTrainer(
        special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],
        vocab_size=50265,
        continuing_subword_prefix="\u0120",
        min_frequency=2,
    )
    # t = AutoTokenizer.from_pretrained("microsoft/deberta-base")
    files = glob.glob("cleaned_train_data/*.csv")
    files_sample = random.choices(files, k=250)
    tokenizer.train(
        files=files_sample,
        trainer=trainer,
    )
    os.makedirs("bpe_tokenizer_0903", exist_ok=True)
    tokenizer.save("bpe_tokenizer_0903")
```

The problem is that the DebertaTokenizer from transformers needs a different set of files to the ones Tokenizers generate. It's ironic that it's also a Huggingface library, because there doesn't seem to be much integration between the 2. Well, as this was the case, I tried many things. First, I tried adding added_tokens.json, special_tokens_map.json, vocab.json, vocab.txt, merges.txt... All these files are included in tokenizer.json (the file generated by huggingface/Tokenizers). However, none of those worked. Then, I tried looking at the files that are saved when you load a DebertaTokenizer from microsoft checkpoints, so that I could copy the structure of the saved folder. I tried to do so, but for the bpe_encoder.bin, there were some difficulties. I used my merges for bpe_encoder["vocab"], as the vocab in the Microsoft bpe_encoder.bin seemed to be merges, and in bpe_encoder["encoder"] I put the vocab dict. For the field bpe_encoder["dict_map"], I couldn't replicate it as token frequencies are not saved by Tokenizers, so I invented them with a random number. However, when I try to train with this tokenizer, it throws a KeyError on step 5, which is strange because when I try to tokenize that concrete token: 'Ŀ', it does indeed tokenize it (by doing DebertaTokenizer.from_pretrained(my_path)("Ŀ"))...

I think all those problems are caused mainly because there is a complete disconnection between Transformers and Tokenizers library, as the tokenizers trained with Tokenizers are not integrable with Transformers, which doesn't make much sense to me, because Tokenizers is supposed to be used to train Tokenizers that are later used in Transformers...

Could please anyone tell me how can I train a Deberta Tokenizer that is, from the beginning, saved with the files needed by Transformers DebertaTokenizer?? Is there any version of Tokenizers in which, when you train a BPETokenizer, it saves the files required by Transformers? 

Thank you very much.


## Expected behavior

It is expected that if 2 libraries are from the same company and the mission of one of the two is to build tools that are later used by the other, the 2 libraries expect and produce the same objects for the same tasks, as it doesn't make sense that you can train a BPETokenizer that you cannot later use as a tokenizer in Transformers. So, what is expected is that if DebertaTokenizer uses BPETokenizer, and this tokenizer expects to receive bpe_encoder.bin, special_tokens_map.json and tokenizer_config.json, then when you train a BPETokenizer with Tokenizers library it should save those objects, not a tokenizer.json file that is useless for later use in Transformers library. 
The tokenizers library powers the fast tokenizers, not the Python "slow" tokenizers. As there is no fast tokenizer for deberta, you can't use the tokenizers library for that model.

You can check which tokenizers have a version backed by the Tokenizers library in [this table](https://huggingface.co/transformers/index.html#bigtable).<|||||>Then, how could we convert the "fast" BPETokenizer to the "slow" BPETokenizer used by Deberta? @sgugger <|||||>Another important thing. I have checked the table and it says that Roberta is able to use fast tokenizer. Deberta, as stated in the paper, uses exactly the same tokenizer as Roberta, so the obvious question is: if Deberta uses Roberta tokenizer, and Roberta tokenizer can be used in "fast" mode, why cannot Deberta be used in "fast" mode??<|||||>Another issue is that when I try to use the BPE Tokenizer trained with huggingface/tokenizers with Roberta directly, it works:

```{python}

tok = RobertaTokenizer.from_pretrained("bpe_tokenizer_0903", use_fast=True)

```

However, when I try to use this same tokenizer for training a language model, it fails:

```{bash}
python -u  transformers/examples/language-modeling/run_mlm_wwm.py \
    --model_type deberta \
    --config_name ./bpe_tokenizer_0903/config.json \
    --tokenizer_name ./bpe_tokenizer_0903 \
    --train_file ./prueba_tr.txt \
    --validation_file ./final_valid.txt  \
    --output_dir ./roberta_1102 \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --evaluation_strategy steps \
    --per_device_train_batch_size 1  \
    --per_device_eval_batch_size 2 \
    --gradient_accumulation_steps 2  \
    --learning_rate 6e-4  \
    --save_steps 10  \
    --logging_steps 10 \
    --overwrite_cache \
    --max_seq_length 128 \
    --eval_accumulation_steps 10 \
    --load_best_model_at_end \
    --run_name deberta_0902 \
    --save_total_limit 10 --warmup_steps 1750  \
    --adam_beta2 0.98 --adam_epsilon 1e-6 --weight_decay 0.01 --num_train_epochs 1

```

The error message is the following:

```
Traceback (most recent call last):
  File "transformers/examples/language-modeling/run_mlm_wwm.py", line 399, in <module>
    main()
  File "transformers/examples/language-modeling/run_mlm_wwm.py", line 286, in main
    use_fast=model_args.use_fast_tokenizer,
  File "/home/alejandro.vaca/data_rigoberta/transformers/src/transformers/models/auto/tokenization_auto.py", line 401, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/alejandro.vaca/data_rigoberta/transformers/src/transformers/tokenization_utils_base.py", line 1719, in from_pretrained
    resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs
  File "/home/alejandro.vaca/data_rigoberta/transformers/src/transformers/tokenization_utils_base.py", line 1790, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/alejandro.vaca/data_rigoberta/transformers/src/transformers/models/roberta/tokenization_roberta_fast.py", line 173, in __init__
    **kwargs,
  File "/home/alejandro.vaca/data_rigoberta/transformers/src/transformers/models/gpt2/tokenization_gpt2_fast.py", line 145, in __init__
    **kwargs,
  File "/home/alejandro.vaca/data_rigoberta/transformers/src/transformers/tokenization_utils_fast.py", line 87, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: data did not match any variant of untagged enum ModelWrapper at line 1 column 1138661
```

Why doesn't it fail when I try to load the tokenizer with RobertaTokenizer.from_pretrained() but it does fail when I try to run run_mlm_wwm.py ? @sgugger @patrickvonplaten @LysandreJik <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.