80min training time to fine-tune BERT-base on the SQuAD dataset instead of 24min?
I just fine-tuned BERT-base on the SQuAD dataset with an AWS EC2 `p3.2xlarge` Deep Learning AMI with a single Tesla V100 16GB:

I used the config in your README:

```
export SQUAD_DIR=/path/to/SQUAD

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```

It took 80min. According to your README:

> This example code fine-tunes BERT on the SQuAD dataset. It runs in 24 min (with BERT-base) or 68 min (with BERT-large) on a single tesla V100 16GB.

How to explain this difference? Is there any way to accelerate the training to 24min as well? Thanks

You should use 16bit training (`--fp16` argument). You can use the dynamic loss scaling or tune the loss scale yourself if the results are not the best.<|||||>@thomwolf Thanks! I enabled 16bit training and it took about 20min/epoch. Is that what you experienced?<|||||>Sounds good.<|||||>@thomwolf 
May I know what is the expected EM & F1 score if users train for 2-3 epochs? I got 43 and 48 respectively.<|||||>You can have a look at the readme examples but it should be a lot higher, around 88-90.
Maybe your batch size is too small, look at the readme for more information.