Thank you for your reply.
I am a bit confused. I think the loss is independent of the number of batchsizes, and the batchsize increases, making the gradient more robust.
In my case, for nn.DataParallel, loss is divided by 32(the total batchsize of 4 gpus is 32); for nn.DistributedDataParallel, the loss of single process is divided by 8(per gpu batchsize is 8), at this time, their gradients are the same, even if the average of the gradient is calculated later(divided by number of precesses), the gradient is almost the same as the former.