Low SQuADv2 F1 & EM Score
Hi,

I used the default settings and run_squad.py script (with the exception of a batch size of 4 since my GPU has low memory) to train for 3 epochs. Turns out I got an EM & F1 score of 43% and 48% respectively. AvNA looks decent at 65%. Is this due to a small number of epochs or the small batch size? Note that training for 3 epochs on my single 1070-TI took 12 hours.

Thanks.
@thomwolf 
Hi, it seems there is something wrong with the training code in this repo.
I used Google's official BERT training code and I could get decent results even with a small batch size: 
`{'EM': 73.80717341230668, 'F1': 77.11048422305339, 'AvNA': 80.78315235274762}`

Here's the settings I used for Google's and this repo's:
```
--vocab_file=uncased_L-12_H-768_A-12/vocab.txt
--bert_config_file=uncased_L-12_H-768_A-12/bert_config.json
--init_checkpoint=uncased_L-12_H-768_A-12/bert_model.ckpt
--do_train=True
--train_file=../squad/data/train-v2.0.json
--do_predict=True
--predict_file=../squad/data/dev-v2.0.json
--train_batch_size=5
--learning_rate=3e-5
--num_train_epochs=2.0
--max_seq_length=384
--doc_stride=128
--output_dir=gugel_bert
--version_2_with_negative=True
--do_lower_case=True
```
```
--bert_model=bert-base-uncased
--output_dir=try_bert_2
--train_file=data/train-v2.0.json
--predict_file=data/dev-v2.0.json
--do_train
--do_predict
--do_lower_case
--train_batch_size=5
--predict_batch_size=5
--num_train_epochs=2.0
--learning_rate=3e-5
--version_2_with_negative
```
EDIT:
Forgot to say that I also can get the expected results using SQuAD v1.1 with this repo's code.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
