Training time increased from 45 min per epoch to 6 hours per epoch in colab 
## 📚 Migration

<!-- Important information -->

Model I am using (Bert): PyTorch-pretrained-Bert and BERT in pytorch-transformers

Language I am using the model on : English

The problem arise when using:
*  the official example scripts: run_squad.py in pytorch-transformers

The tasks I am working on is:
* an official GLUE/SQUaD task: Fine Tuning SQuAD

Details of the issue:

Hi finetuned model over SQuAD with the following code in colab:
%run run_squad.py --bert_model bert-base-uncased \
                  --do_train \
                  --do_predict \
                  --do_lower_case \
                  --fp16 \
                  --train_file SQUAD_DIR/train-v1.1.json  \
                  --predict_file SQUAD_DIR/dev-v1.1.json \
                  --train_batch_size 12 \
                  --learning_rate 3e-5 \
                  --num_train_epochs 2.0 \
                  --max_seq_length 384 \
                  --doc_stride 128 \
                  --output_dir debug_squad9

The above code took 45 min for one epoch with  "exact_match": 81.94891201513718, "f1": 89.02481046726041. I ran this code(PyTorch-pretrained-Bert) somewhere between July 20th - 27th 2019.

And now the same file with the above code(PyTorch-pretrained-Bert) is taking the 6 hours for one epoch. Why is that?  

I have tested with pytorch-transformers as well, It is also taking 6 hours for one epoch.
 
I am unable to understand. Is there any change with apex or implementation ??
 
## Environment

* OS: google colab
* Python version: Python 3.6.8
* PyTorch version: 1.2
* PyTorch Transformers version (or branch): the existing version pytorch-transformers
* Using GPU ? - Yes I guess K80 single GPU
* Distributed of parallel setup ? No
* Any other relevant information:
- The above environment details on based on the exiting run which took 6 hours for one epoch.
- I haven't collected the environment information when I tested it for the first time over PyTorch-pretrained-Bert.

## Checklist

- [yes ] I have read the migration guide in the readme.
- [yes ] I checked if a related official extension example runs on my machine.

## Additional context

Is there anything new that I am missing from pytorch-Transformers or implementation change in PyTorch-pretrained-Bert ??
Hi, 

Found a solution to my problem to some extent. I had cloned the latest apex repo and testing out with PyTorch-pretrained-Bert which is causing the problem to take more time for execution. I took the older apex repo and tested the code and working as earlier (45 min per epoch)

But when I was using the pytorch-transformers with latest apex repo it is taking 6 hours for one epoch. 

Is that usual to take 6 hours for one epoch in colab ??
<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
