run_ner.py fails when loading a model/checkpoint from a directory
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-3.10.0-957.1.3.el7.x86_64-x86_64-with-centos-7.6.1810-Core
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
 --> @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): XLMRobertaModel

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce


Steps to reproduce the behaviour:

The run_ner.py seems to be unable to load models from a directory. The following command works as expected:

```
python transformers/examples/token-classification/run_ner.py --dataset_name conll2003 --model_name_or_path xlm-roberta-base --output_dir output --num_train_epochs 10 --per_device_train_batch_size 32 --per_device_eval_batch_size 32 --learning_rate 5e-05 --seed 29 --save_steps 9223372036854775807 --do_train --do_eval --overwrite_cache --overwrite_output_dir  --fp16
```

However, if we load 'xlm-roberta-base', we save the model to a directory, and we try to run the script with the directory set as the model path, the script fails. (Same behaviour if we use XLMRobertaForTokenClassification instead of XLMRobertaModel in step 1)

1.

```
from transformers import XLMRobertaTokenizer,  XLMRobertaModel
tokenizer =  XLMRobertaTokenizer.from_pretrained('xlm-roberta-base')
model =  XLMRobertaModel.from_pretrained('xlm-roberta-base')
spath = "models/xlmroberta"
tokenizer.save_pretrained(spath)
model.save_pretrained(spath)
```


2.
```
python transformers/examples/token-classification/run_ner.py --dataset_name conll2003 --model_name_or_path models/xlmroberta --cache_dir models/xlmroberta --output_dir output --num_train_epochs 10 --per_device_train_batch_size 32 --per_device_eval_batch_size 32 --learning_rate 5e-05 --seed 29 --do_train --do_eval --overwrite_cache --overwrite_output_dir
```

Error message (Running the command in CPU to be able to see it): 

```
INFO|tokenization_utils_base.py:1688] 2021-02-07 14:38:00,146 >> Model name 'models/xlmroberta' not found in model shortcut name list (xlm-roberta-base, xlm-roberta-large, xlm-roberta-large-finetuned-conll02-dutch, xlm-roberta-large-finetuned-conll02-spanish, xlm-roberta-large-finetuned-conll03-english, xlm-roberta-large-finetuned-conll03-german). Assuming 'models/xlmroberta' is a path, a model identifier, or url to a directory containing tokenizer files.
[INFO|tokenization_utils_base.py:1721] 2021-02-07 14:38:00,146 >> Didn't find file models/xlmroberta/tokenizer.json. We won't load it.
[INFO|tokenization_utils_base.py:1721] 2021-02-07 14:38:00,147 >> Didn't find file models/xlmroberta/added_tokens.json. We won't load it.
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,147 >> loading file models/xlmroberta/sentencepiece.bpe.model
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file None
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file None
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file models/xlmroberta/special_tokens_map.json
[INFO|tokenization_utils_base.py:1784] 2021-02-07 14:38:00,148 >> loading file models/xlmroberta/tokenizer_config.json
[INFO|modeling_utils.py:1025] 2021-02-07 14:38:02,120 >> loading weights file models/xlmroberta/pytorch_model.bin
[INFO|modeling_utils.py:1143] 2021-02-07 14:38:13,379 >> All model checkpoint weights were used when initializing XLMRobertaForTokenClassification.

[WARNING|modeling_utils.py:1146] 2021-02-07 14:38:13,380 >> Some weights of XLMRobertaForTokenClassification were not initialized from the model checkpoint at models/xlmroberta and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
100%|###################################################################################################################################################################################################################################################################################################################################################################################################| 15/15 [00:01<00:00,  7.55ba/s]
100%|#####################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [00:00<00:00,  8.69ba/s]
100%|#####################################################################################################################################################################################################################################################################################################################################################################################################| 4/4 [00:00<00:00,  8.25ba/s]
[INFO|trainer.py:429] 2021-02-07 14:38:18,120 >> The following columns in the training set don't have a corresponding argument in `XLMRobertaForTokenClassification.forward` and have been ignored: tokens, ner_tags, chunk_tags, id, pos_tags.
[INFO|trainer.py:429] 2021-02-07 14:38:18,121 >> The following columns in the evaluation set don't have a corresponding argument in `XLMRobertaForTokenClassification.forward` and have been ignored: tokens, ner_tags, chunk_tags, id, pos_tags.
[INFO|trainer.py:721] 2021-02-07 14:38:18,122 >> Loading model from models/xlmroberta).
[INFO|configuration_utils.py:443] 2021-02-07 14:38:18,123 >> loading configuration file models/xlmroberta/config.json
[INFO|configuration_utils.py:481] 2021-02-07 14:38:18,125 >> Model config XLMRobertaConfig {
  "architectures": [
    "XLMRobertaModel"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "xlm-roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "transformers_version": "4.3.0.dev0",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 250002
}

[INFO|modeling_utils.py:1025] 2021-02-07 14:38:18,126 >> loading weights file models/xlmroberta/pytorch_model.bin
[INFO|modeling_utils.py:1143] 2021-02-07 14:38:30,381 >> All model checkpoint weights were used when initializing XLMRobertaForTokenClassification.

[WARNING|modeling_utils.py:1146] 2021-02-07 14:38:30,381 >> Some weights of XLMRobertaForTokenClassification were not initialized from the model checkpoint at models/xlmroberta and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[INFO|trainer.py:832] 2021-02-07 14:38:30,394 >> ***** Running training *****
[INFO|trainer.py:833] 2021-02-07 14:38:30,394 >>   Num examples = 14041
[INFO|trainer.py:834] 2021-02-07 14:38:30,394 >>   Num Epochs = 10
[INFO|trainer.py:835] 2021-02-07 14:38:30,394 >>   Instantaneous batch size per device = 32
[INFO|trainer.py:836] 2021-02-07 14:38:30,394 >>   Total train batch size (w. parallel, distributed & accumulation) = 32
[INFO|trainer.py:837] 2021-02-07 14:38:30,395 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:838] 2021-02-07 14:38:30,395 >>   Total optimization steps = 4390
  0%|                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | 0/4390 [00:00<?, ?it/s]Traceback (most recent call last):
  File "third_party/transformers/examples/token-classification/run_ner.py", line 454, in <module>
    main()
  File "third_party/transformers/examples/token-classification/run_ner.py", line 388, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/trainer.py", line 931, in train
    tr_loss += self.training_step(model, inputs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/trainer.py", line 1295, in training_step
    loss = self.compute_loss(model, inputs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/trainer.py", line 1325, in compute_loss
    outputs = model(**inputs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 1349, in forward
    loss = loss_fct(active_logits, active_labels)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 962, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/ikerlariak/igarcia945/envs/transformers422/lib/python3.6/site-packages/torch/nn/functional.py", line 2264, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 3 is out of bounds.
```


Same [issue](https://github.com/huggingface/transformers/issues/10051) as me 😄 <|||||>I'm guessing this has to do with the number of labels your model has. When doing the following:

```py
model =  XLMRobertaModel.from_pretrained('xlm-roberta-base')
spath = "models/xlmroberta"
model.save_pretrained(spath)
```

you're saving a model to disk that has no classification head, and the script will therefore use the default number of labels when loading it. I would advise you do the following instead, with `NUM_LABELS` your number of labels:

```py
model =  XLMRobertaForTokenClassification.from_pretrained('xlm-roberta-base', num_labels=NUM_LABELS)
spath = "models/xlmroberta"
model.save_pretrained(spath)
```

Please let me know if this fixes your issue.<|||||>@LysandreJik problem solved. Thank you!!