I find that BatchNorm has similar behavior. When using the DataParallel, BatchNorm update running mean and running variance at each replica. However, each replica free the running mean and running variance buffer at the end of the iteration. Moreover, weights and buffers of the replica on device[0] share storage with those of the input model! Therefore, BatchNorm only uses the batch statistics on device[0], and DataParallel will replicate the weight and buffer to other GPUs.