run_squad script gets stuck
Hello, 

I am trying to run the squad fine tuning script, but it hangs after printing out a few predictions. I am attaching the log. Can you help take a look?

I am running the script on a machine with 8 M40s.

[bert_squad.log](https://github.com/huggingface/pytorch-pretrained-BERT/files/2634588/bert_squad.log)


Best,
Samyam
Never mind, it just needed time to process the examples. It might be good to have the progress bar inside convert_examples_to_features.<|||||>Maybe try distributed training? I don't think PyTorch `DataParallel` will be very efficient on 8 GPUs due to the python GIL.<|||||>Thanks for the suggestion. I will try that. Currently, its showing me about 9 hours to fine tune bert-large on squad with batch size of 32 using DataParallel.

The performance improves quite a bit if a if I use a batch size of 256 with gradient accumulate, which makes sense as this reduces the frequency of communication of the gradients. A question I have is, does the learning rate adapt automatically to the batch size being used?  Have you tried larger batch sizes?