when using multiple GPUs, `loss.mean()` may have subtle bias 
The problem is that, when the input is distributed to multiple GPUs, the input on each GPU may have different `batch_size`. 

For example, if you have 2 GPUs and the total batch_size is 13, then the `batch_size` for each GPU will be 7 and 6 respectively, `loss.mean()` will not give the exact loss. Although it may have little influence on the training of the model, it is not the exact result.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L1006-L1007
You could fix it with something like this
```
batch_size = torch.tensor(data.shape[1]).to(device)
dist.all_reduce(batch_size, op=dist.ReduceOp.SUM)
dist.all_reduce(loss, op=dist.ReduceOp.SUM)
mean_loss = loss/batch_size
```<|||||>Thanks for your solution. I don't think it could fix it, because the `loss` returned by each GPU is already averaged over its `batch_size`.<|||||>@Lvzhh the examples (like `run_squad.py`) only work in two settings for multi-gpu:

- using `DataParallel` (not distributed, triggered when you simply run the script on a multi-gpu server): in this case the batch size should be a multiple of the number of gpu or an error is thrown. This is the case in which this `loss = loss.mean()` will be triggered. No risk of differing batch sizes here.
- using `DistributedDataParallel` with a *single* GPU assigned to each process (triggered when you run the example using PyTorch `distributed.launch` script for instance): this only works when 1 GPU is assigned to each process (see [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/examples/run_squad.py#L853-L857)). In this case `loss = loss.mean()` is not triggered. There is no averaging of the loss over the GPU. What happens here is that it's the gradients which are averaged (read PyTorch [DDP doc](https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel) for instance). The batch size should be the same for each script so no problem here also.

So I don't think the problem you are describing will show up in practice.
<|||||>@thomwolf I'm using the first multi-gpu setting `DataParallel`. And I tried to find some code in `run_squad.py` to ensure that the `batch_size` be a multiple of the number of gpu, but I didn't find.

And I just run this command [here](https://github.com/huggingface/pytorch-pretrained-BERT#squad) with `--train_batch_size 13` using 2 GPUs, and print the size of `start_logits` and its device, the first dimension (`batch_size`) on each GPUs are actually different (7 vs 6).

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3fc63f126ddf883ba9659f13ec046c3639db7b7e/pytorch_pretrained_bert/modeling.py#L1204-L1206<|||||>Oh you are right, this error is not there anymore.
We should probably just check for this again, I'll fix the examples.

`DataParallel` is in the process of being replaced by `DistributedDataParallel` which is pretty much always faster due to releasing the GiL so maybe you should try DDP.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
