In that case, when computing the local gradient, the functions, inputs, and function values are exactly same as the case of the single processor platform.

This is actually not true. Say we have a function f(x) = w * x, where w is the weight. Then when you compute gradient (i.e., dw), you will need both df (from loss, which depends on local input) and x (from local input or intermediate local output, which also depend on local input). So, if not communicating gradients, we need to communicate both the final loss and the intermediate outputs of all layers.



 TT_YY:

does averaging the local gradients computed from the local losses produce exactly the same one as the global gradient computed from the global loss?


No, this is not guaranteed to be the same, but due to a different reason. If 1) the loss function satisfies the condition loss_fn([x1, x2]) == (loss_fn(x1) + loss_fn(x2)) / 2 and 2) batch size on all processes are the same, then average gradients should be correct. Otherwise, average won’t produce the same result. One example would be, if we use .sum() as the loss function, we should just sum instead of averaging the gradient.

If the answer is no, I think that we need to average the local losses to produce a global loss and hand it to all the GPUs to compute correct local gradients that are averaged to produce a correct global gradient. This might be achieved by performing all_reduce() over the local losses  before  doing loss.backward() on each GPU.

I might miss sth. If we do the above, it means we compute the gradients using global loss and local activation (i.e., global df and local x in the f(x)=w*x example above). In this case, what does this gradient mean?