Solved by GeoffreyChen777 in post #11 
              
              
                @Lausanne I think you should keep the original learning rate. 
If you use the DistributedDataParallel, the gradient will be averaged between each process. DataParallel sum the gradient. It is equal to the DistributedDataParallel. The reason is that the the loss will be averaged by 128 batchsize and …