Thank you again.
Maybe you have fully answered my question, but I still feel that my point is missing. As I understand, a gradient is computed by the back propagation using the chain rule and first derivative of functions in a model network. Also, as you mentioned, we need the function vales within the network, as well as the loss.
Since the method existed far before the parallelism era, the back-prop naturally started from a single “total” or “global” loss in the single processor platform. Therefore, in that case, we use a loss readily averaged over a batch of input. On the other hand, in the multi-GPU platform, a batch input is farther divided into smaller batches each of which is used to produce a “local” loss by a GPU. In that case, when computing the local gradient, the functions, inputs, and function values are exactly same as the case of the single processor platform. Only difference is using the local loss instead of the global loss.
My question is; does averaging the local gradients computed from the local losses produce exactly the same one as the global gradient computed from the global loss?
If the answer is no, I think that we need to average the local losses to produce a global loss and hand it to all the GPUs to compute correct local gradients that are averaged to produce a correct global gradient. This might be achieved by performing all_reduce() over the local losses before doing loss.backward() on each GPU.
The answer could be yes, but I don’t know the mathematical explanation for it.
That is my point.
If I misunderstand something, please point it out. Thank you.