Cannot Evaluate While Training Using the Trainer
@sgugger 

## Environment info
   
- `transformers` version: 4.0.0
- Platform: AWS Amazon Linux 
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


## Information

Model I am using (Bert, XLNet ...): Third Party Model ( Routing Transformer)

The problem arises when using:
* [ ] my own modified scripts:
I am using a custom Trainer to train the third party model. The training loops runs smoothly without evaluation. But as soon as I try to do evaluation while training, it stops training after a couple of evaluations even though it has not reached the max_steps. I copy my args and trainer below.  I understand in previous versions there was a evaluate_during_training flag that some have suggested as a fix until Sep 20. But that flag doesn't seem to exist anymore. Any help or pointers would be appreciated. 


The tasks I am working on is:
* [ ] my own task or dataset: Masked Language Modeling 

## To reproduce

Steps to reproduce the behavior:

```
custom_args = MLMArguments(
        output_dir='../models/', 
        mask_ratio=0.2, 
        do_train=True,
        do_eval=True,
        max_steps=2000000, 
        save_steps=50,
        logging_steps=5,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        logging_dir = '../logs/',
        evaluation_strategy="steps",
        prediction_loss_only=True

)

checkpoint_callback=TrainerCallback()
tb_callback = TensorBoardCallback()

custom_trainer = MLMTrainer(
        rt_model, 
        args=custom_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        callbacks=[checkpoint_callback, tb_callback]
        
)

custom_trainer.train()
```


Step | Training Loss | Validation Loss
-- | -- | --
5 | 10.033914 | 10.017973
10 | 10.028201 | 9.935969

and at this point it prints the output and stops training.
Hi there, I would like to help you, but the code you are providing is not runable on my side (MLMTrainer, MLMArguments, train_dataset, eval_dataset and rt_model for instance are not defined). Could you please post a complete and short reproducer of the bug?