Weird PanicException when trying to save tokenizer using `tokenizer.save_pretrained`
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Using accelerate but there's only 1 GPU currently 

### Who can help

@LysandreJik, @SaulLu, @sgugger 

 -->

## Information

Model I am using (Bert, XLNet ...): YituTech/conv-bert-base

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below) . Modified scripts but save_model method which is causing issues is exact same as this: https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below) - Similar GLUE like paired sentence classification task

## To reproduce

Steps to reproduce the behavior:

For some models such as the one mentioned above, I've seen some times the script abruptly fails with a weird PanicException while trying to save the tokenizer. I don't think this is model specific issue, because other 3p models have saved successfully for me always, so looks like something to do with Transformers/tokenizers lib. See code that I'm using to save the model below, along with detailed logs as well. 

I've no idea about this issue, and how to fix it. For other standard models like bert, roberta etc the saving has worked fine always, along with some other 3p models that I've tried as well. Please let me know if you any need more information from me on this? 

I'm saving both the model and tokenizer only in main_process, I hope that's right? I find this slightly weird because clearly if you see from the logs below model is getting saved to dir successfully, but tokenizer specifically is failing. Not sure what could be the peculiarity here causing this? Any suggestions that you might have to fix this would be really helpful! 

Relevant code to save model, which is causing the issue - same as this: https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/run_glue_no_trainer.py#L460 . I've also included a couple of lines at the top in the code blurb below to show how I'm initializing the config/tokenizer/model if that might be of any relevance:
```
    config = AutoConfig.from_pretrained(args.model_name_or_path, num_labels=num_labels, finetuning_task=None)    
    tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, use_fast=not args.use_slow_tokenizer)
    model = AutoModelForSequenceClassification.from_pretrained(
        args.model_name_or_path,
        from_tf=bool(".ckpt" in args.model_name_or_path),
        config=config,
        cache_dir=args.output_dir
    )

    def save_model(model, tokenizer, args):
        if args.output_dir is not None and accelerator.is_main_process:
            # Handle the repository creation
            if args.output_dir is not None:
                os.makedirs(args.output_dir, exist_ok=True)
            accelerator.wait_for_everyone()
            unwrapped_model = accelerator.unwrap_model(model)
            unwrapped_model.save_pretrained(args.output_dir, save_function=accelerator.save)
            tokenizer.save_pretrained(args.output_dir)
```

Logs: 

```
03/18/2022 11:51:47 - INFO - __main__ - ***** Running training *****
03/18/2022 11:51:47 - INFO - __main__ -   Num examples = 25227
03/18/2022 11:51:47 - INFO - __main__ -   Num Epochs = 10
03/18/2022 11:51:47 - INFO - __main__ -   Instantaneous batch size per device = 32
03/18/2022 11:51:47 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 32
03/18/2022 11:51:47 - INFO - __main__ -   Gradient Accumulation steps = 1
03/18/2022 11:51:47 - INFO - __main__ -   Total optimization steps = 7890
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [28:11<00:00,  6.46it/s]03/18/2022 12:20:32 - INFO - __main__ - *** Predict ***
Configuration saved in conv-bert-base/config.json
Model weights saved in conv-bert-base/pytorch_model.bin
tokenizer config file saved in conv-bert-base/tokenizer_config.json
Special tokens file saved in conv-bert-base/special_tokens_map.json
thread '<unnamed>' panicked at 'no entry found for key', /__w/tokenizers/tokenizers/tokenizers/src/models/mod.rs:36:66
Traceback (most recent call last):
  File "run.py", line 566, in <module>
    main()
  File "run.py", line 563, in main
    tokenizer.save_pretrained(args.output_dir)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 2108, in save_pretrained
    filename_prefix=filename_prefix,
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 599, in _save_pretrained
    self.backend_tokenizer.save(tokenizer_file)
pyo3_runtime.PanicException: no entry found for key
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7890/7890 [29:26<00:00,  4.47it/s]
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/accelerate_cli.py", line 41, in main
    args.func(args)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py", line 384, in launch_command
    simple_launcher(args)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/accelerate/commands/launch.py", line 142, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ec2-user/anaconda3/envs/pytorch_latest_p36/bin/python', 'run.py', '--model_name_or_path', 'YituTech/conv-bert-base', '--train_file', '../data/train.csv', '--validation_file', '../data/val.csv', '--test_file', '../data/test.csv', '--max_length', '128', '--per_device_train_batch_size', '32', '--learning_rate', '2e-5', '--num_train_epochs', '10', '--output_dir', 'conv-bert-base/']' returned non-zero exit status 1.
```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Tokenizer save should be successful. 

It looks like your issues comes from a problem with the fast tokenizer. A simple reproducer is:
```py
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("YituTech/conv-bert-base")
tokenizer.save_pretrained("test-tok")
```<|||||>Ohh, what should I do in that case - should i use the slow one? More specifically, why is the fast tokenizer failing only with this model and not others - not sure if it could be something model specific? @sgugger <|||||>It's probably linked to this specific architecture. Using the slow tokenizer is a workaround, I've pinged @Narsil in private to look at the fast tokenizer :-)<|||||>Thanks look forward to the findings, this issue has been causing me some trouble off late :(. I can't recollect the exact model name now, but I remember briefly seeing this same error for some other models too earlier. I initially thought it might have been some transient error, but looks like clearly it is not as it's been reproduced multiple times off late for me. @sgugger <|||||>Some ablation experiments were going in the background for me, so just noticed: This failed for YituTech/conv-bert-small too btw with the exact same exception. And assuming YituTech/conv-bert-medium-small this would also fail similarly (haven't tested it explicitly). Very weird. Hopefully there's something that can be fixed here for the fast_tokenizer. <|||||>Hi @ashutoshsaboo ,

The fix for the fast tokenizers is here: https://github.com/huggingface/tokenizers/pull/954

The main issue, is that those tokenizers actually contain duplicate tokens. Saving the using the `slow` tokenizer will trigger a bunch of warnings. `tokenizers` will follow the same path.

Please keep in mind that this means that this models underuse its capacity (I counted 1008 tokens are purely ignored, taking up space in the embedding matrix while never being used).

We'll r<|||||>Woah that was quick to get the fix out, thanks so much first of all! What do you mean by "tokens are purely ignored"? Aren't tokens dependent on vocabulary end of the day with some special extra specific tokens like SEP, PAD, CLS etc, or did I infer it wrong? @Narsil <|||||>Hi @ashutoshsaboo ,

> Aren't tokens dependent on vocabulary end of the day

Yes, but for these files, the `vocab.txt` contains duplicate (look at the last few lines for instance).
The problem about those duplicates is that the vocabulary expects one token per line, and here we have duplicates. The way `transformers` and `tokenizers` work in that instance, is the overwrite the previous token when they are duplicates.

The net-effect means that the ids of the tokens that get overwritten, will never show up in you tokenization, meaning they will never get used by your model, and you are using the weight space in your model (in the `embeddings` table) even though you are never going to use it. It's not critical as the model will probably work, but it's a waste of parameter space. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Closing since this has been fixed.