Why average the loss when training on multi-GPUs
Could anyone help to explain the motivation for this operation?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L573-L574
Multi-GPU loss returns a tuple of losses with one loss for each GPU.
We average them to get the full loss. See [this blog post](https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255) for more details.