@Lausanne I think you should keep the original learning rate.
If you use the DistributedDataParallel, the gradient will be averaged between each process. DataParallel sum the gradient. It is equal to the DistributedDataParallel. The reason is that the the loss will be averaged by 128 batchsize and then backward to DataParallel model. The gradient will be reduced across each minibatch. And in DistributedDataParallel, the loss will be averaged by 32 and backward to each distributed model. So, we need to average the gradients between each distributed process.