DistilBERT for token classification (pytorch) predicts wrong classes for <PAD> tokens
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-47-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@stefan-it @LysandreJik 

## Information

Model I am using: DistilBERT (pytorch), but I guess, the same is true for BERT.

The problem arises when using:
* my own modified scripts: (give details below)

The tasks I am working on is:
* my own task or dataset: (give details below)

## To reproduce
This is NER task on specially compiled dataset which I cannot provide, but it is not that much different from CoNLL.
I tried DistillBERT tensorflow and pytorch versions. Input is `inputs_ids` and `attention_mask` in both cases.
Results for DistillBERT tensorflow:
![image](https://user-images.githubusercontent.com/22475959/93342115-9d0d3680-f837-11ea-8b0b-5001856bd1e7.png)
model was compiled as:
```
model = TFDistilBertForTokenClassification.from_pretrained(MODEL, num_labels=len(LABELS))
model.layers[-1].activation = tf.keras.activations.softmax
optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
```
code for report (I didn't remove prediction for padding from the model output as I assumed that model should not have any difficulties predicting it as `O`. And it does in this case)
```
predictions_classes = [LABELS_MAP[_id] for _id in predictions_normalized.reshape(-1)]
true_classes = [LABELS_MAP[_id] for _id in label_ids_test.reshape(-1)]

print(
    classification_report(
        true_classes, 
        predictions_classes,
        labels=sorted(list(set(true_classes) - {'O'}))
    )
)
```

Due to some issues with conversion to onnx format, I had to create pytorch version of the same model. Dataset split is the same in both versions.
Here is output of DistillBERT pytorch:
![image](https://user-images.githubusercontent.com/22475959/93352453-879e0980-f843-11ea-883f-9207fbfe603a.png)
Notice that support is the same as above, because it is the same data. Recall is comparable to tf. Precision is way off.
Model is compiled as:
```
training_args = TrainingArguments(
    output_dir='results',          
    num_train_epochs=4,              
    per_device_train_batch_size=4,  
    per_device_eval_batch_size=32,   
    warmup_steps=0, 
    weight_decay=0, 
    logging_dir=None,           
    logging_steps=100,
    eval_steps=500,
    save_steps=1e6,
    learning_rate=1e-5,
    seed=8,
    evaluate_during_training=True,
    do_eval=True,
    disable_tqdm=True,
)

model = DistilBertForTokenClassification.from_pretrained('distilbert-base-multilingual-cased', num_labels=len(LABELS))

adam = Adam(model.parameters(), lr=1e-5)

trainer = Trainer(
    model=model,                         
    args=training_args,                  
    train_dataset=train_dataset,         
    eval_dataset=test_dataset,
    optimizers=(adam, get_constant_schedule(adam))             
)

trainer.train()
```
The reason for huge drop in precision is that model predicts classes other than `O` for pad tokens, because their loss is ignored in loss function:
![image](https://user-images.githubusercontent.com/22475959/93343751-8b2c9300-f839-11ea-949b-a7a36f4a1342.png)

If I remove predictions for all special tokens then result looks like:
![image](https://user-images.githubusercontent.com/22475959/93352634-b7e5a800-f843-11ea-90a7-a1540c33cbad.png)
Now, it looks like results from tf version.

If I comment out these lines:
https://github.com/huggingface/transformers/blob/9e376e156a78aa08f802d569d829064aff930c58/src/transformers/modeling_distilbert.py#L822-L830
and keep only 
`loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))` taking into account loss for pad tokens and still using attention mask as the second input than model doesn't predict PAD tokens as classes other than `O` anymore. Output:
![image](https://user-images.githubusercontent.com/22475959/93354973-40654800-f846-11ea-88a1-47203adcaa67.png)

So, performance definitely didn't suffer and model stopped predicting classes other than `O` for PAD tokens.
![image](https://user-images.githubusercontent.com/22475959/93355153-6b4f9c00-f846-11ea-8f9a-f75e43da6fb3.png)


## Expected behavior
I think, SOTA models shouldn't predict classes other than `O` for PAD tokens (or some special class). 

The [PAD] tokens are not used for loss calculation if I remembered correctly, and therefore its predictions has no meaning. You should just ignore the [PAD] tokens results.

Put [PAD] tokens into loss calculation will make the model biased toward  [PAD] tokens, which is undesirable.