I used distributeddataparallel, does this method sum or average the gradients of multi-gpus?
I think this settings will have influence on adjusting the learning rate. According to the learning rate linear rule, when batch size is double, the learning rate should also be doubled. If the gradients are summed, the amplitude of the gradients are thus double, which means that I no longer need to double the learning rate, and I only need to adjust the learning rate according to the change of batch size on one single gpu, rather than the change of batch size brought by increasing gpu numbers. Would you please give some suggestions on how to adjust learning rate in the multi-gpu mode?