@teng-li Thank you!
I am reproducing a huge network. The author use 16 batch size with 4GPU training. And they use DataParallel. I don’t have 4GPU machine. So I want to use 2 machine(2GPUs on each machine) to traing the network with batchsize 16. If the averaging is the default operation of DisributedDataParallel, there is no way to reproduce the training process. Right?