Stacked Roberta run_mlm.py
# 🖥 Benchmarking `transformers`

## Benchmark

I try to run [transformers/experiments/language_modeling/run_mlm.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) in order to train Roberta from scratch on Wikipedia dataset.

## Set-up
Transformers version: 4.4.0.dev0
4 GPUs: NVIDIA Tesla V100 16 GB; 4096 Bit, PCI Express 3.0 x16.
Used bash run script:
```bash
#!/bin/bash

export CUDA_LAUNCH_BLOCKING=1

source /data/env/bin/activate

nohup python3 transformers/examples/language-modeling/run_mlm.py \
--dataset_name wikipedia \
--tokenizer_name roberta-base \
--model_type roberta \
--dataset_config_name 20200501.en \
--do_train \
--do_eval \
--learning_rate 1e-5 \
--num_train_epochs 5 \
--save_steps 5000 \
--warmup_steps=10000 \
--output_dir /data/models/wikipedia_roberta \
&
```
Tested with the script and also directly with python command (without nohup)

## Results

Code seems stacked in `trainer.py`, at the first compute_loss step, when performing inference:
```python
    def compute_loss(self, model, inputs, return_outputs=False):
        """
        How the loss is computed by Trainer. By default, all models return the loss in the first element.

        Subclass and override for custom behavior.
        """
        if self.label_smoother is not None and "labels" in inputs:
            labels = inputs.pop("labels")
        else:
            labels = None
        print("STACKED HERE")
        outputs = model(**inputs)
...
```
I can't understand which running parameters are wrong. Could inference for a single batch take more than 30 mins?
Thanks in advance!

UPDATE: Without --warmup_steps is working.