Hi,
It depends how you do multigpu. If you use DataParallel, then the gradients are accumulated on the first device in the list and then the new weights are shared to all the gpus.
For batchnorm when not synced I am not sure but I would guess it uses the statistics computed on the first device in the list.