SQuAD 1.1 very low evaluation score when using `--fp16`
I'm replicating SQuAD 1.1 https://github.com/huggingface/pytorch-pretrained-BERT/tree/v0.6.2#squad on latest release `v0.6.2`.

My setup: 
* GeForce RTX 2080 Ti
   * Driver Version: 418.43
   * CUDA Version: 10.1
* Linux Ubuntu 18.10
* pytorch 1.1.0 (installed via conda: py3.7_cuda10.0.130_cudnn7.5.1_0)
* latest `apex` package

I'm on latest release:
```
$ git status 
HEAD detached at v0.6.2
```
-------------

I'm replicating fine tuning bert on SQuAD 1.1. When executing without `--fp16` I'm getting expected result:
```
$ python evaluate-v1.1.py dev-v1.1.json /tmp/debug_squad/predictions.json
{"exact_match": 81.58940397350993, "f1": 88.6984251786611}
```

Full command:
```
export SQUAD_DIR=squad_11_data

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```

Same experiment with `--fp16` I'm getting very poor results:
```
$ python evaluate-v1.1.py dev-v1.1.json /tmp/debug_squad_fp16_apex/predictions.json 
{"exact_match": 0.47303689687795647, "f1": 8.678859681492447}
```

Full command:
```
export SQUAD_DIR=squad_11_data

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 8 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad_fp16/ \
  --fp16
```
I remember that I saw information about gradient overflow several times (sorry, don't have more details I have lost output logs).

How to get decent results when using `--fp16`?
I took example code from https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-bert-large-on-gpus (which has additional option `--loss_scale 128`). Still getting very low test scores:
```
$ python evaluate-v1.1.py dev-v1.1.json ../output/debug_squad_fp16/predictions.json 
{"exact_match": 0.5771050141911069, "f1": 8.853750220358535}
```

Is there any known bug in current (on `v0.6.2`) PyTorch BERT implementation or is this my setup?

---

**UPDATE**
Probably there is a bug in current stable `v0.6.2` version. When running same command on latest `master` branch I'm getting good results:
```
$ python evaluate-v1.1.py dev-v1.1.json ../output/debug_squad_fp16/predictions.json
{"exact_match": 81.45695364238411, "f1": 88.71433452234619}
```<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
