issue in pretraining language model with checkpoint 
# 🐛 Bug

## Information

I am pre-training albert from scratch and it was going fine.(8 v100)
But when I am training using a checkpoint its using single gpu and that also 1gb/32gb GPU ram.



## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
 `transformers` version: 2.10.0

launching script with:
```
python transformers/examples/language-modeling/run_language_modeling.py   --train_data_file text.txt  --output_dir albert_model --model_type albert --mlm --config_name test --tokenizer_name test --do_train --line_by_line --learning_rate 5e-5 --num_train_epochs 3 --save_total_limit 50 --save_steps 5000 --per_gpu_train_batch_size 150 --seed 42 --overwrite_output_dir --max_steps 200000 --fp16 --model_name_or_path albert_model/checkpoint-200000
```
It seems that its starting it properly with global step but not using gpu properly which really weird
```
/language_model/lm/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:218: UserWarning: Please also save or load the state of the optimzer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)
Selected optimization level O1:  Insert automatic casts around Pytorch functions and Tensor methods.

Defaults for this optimization level are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O1
cast_model_type        : None
patch_torch_functions  : True
keep_batchnorm_fp32    : None
master_weights         : None
loss_scale             : dynamic
Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback.  Original ImportError was: ModuleNotFoundError("No module named 'amp_C'",)
06/08/2020 11:25:09 - INFO - transformers.trainer -   ***** Running training *****
06/08/2020 11:25:09 - INFO - transformers.trainer -     Num examples = 28236463
06/08/2020 11:25:09 - INFO - transformers.trainer -     Num Epochs = 43
06/08/2020 11:25:09 - INFO - transformers.trainer -     Instantaneous batch size per device = 150
06/08/2020 11:25:09 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 1200
06/08/2020 11:25:09 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/08/2020 11:25:09 - INFO - transformers.trainer -     Total optimization steps = 1000000
06/08/2020 11:25:09 - INFO - transformers.trainer -     Continuing training from checkpoint, will skip to saved global_step
06/08/2020 11:25:09 - INFO - transformers.trainer -     Continuing training from epoch 8
06/08/2020 11:25:09 - INFO - transformers.trainer -     Continuing training from global step 200000
06/08/2020 11:25:09 - INFO - transformers.trainer -     Will skip the first 11752 steps in the first epoch
Epoch:   0%|                                                                                                                               | 0/35 [00:00<?, ?it/s]

Iteration:  42%|███████████████████████████████████████████▋                                                             | 9781/23531 [1:20:59<1:53:51,  2.01it/s]
```
can any one suggest something here?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
