Thanks a lot! I know what you mean. So, in summarize, the multi-gpu environment works like following:

Scatter the model and the state dict from GPU:0 to all the GPUs.
Split the data, and seperately forward them on different GPUs.
Gather output from GPUs to GPU:0
Calculate Loss by using outputs and targets on GPU:0
Backward Loss to GPUs and seperately calculate gradients
Gather gradients from GPUs to GPU:0
Update parameters on GPU:0
GoTo Step1.

So, the only thing that not fully synchronized is the mean and var of BN, because it does not gather to GPU:0 during backward. All the other parameters are fully synchronized because of the gather-scatter mechanism.