Thank you for your further explanation.

So, if not communicating gradients, we need to communicate both the final loss and the intermediate outputs of all layers.

Yes, I agree that we must communicate gradients to have a global gradient. My question is about relationship between the global loss and the local gradients, not about communicating losses instead of gradients.

If 1) the loss function satisfies the condition  loss_fn([x1, x2]) == (loss_fn(x1) + loss_fn(x2)) / 2  and 2) batch size on all processes are the same, then average gradients should be correct.

I understand that, in a parallel process, the losses are locally averaged on a GPU, and the resulting losses can be globally averaged. That is the reason why the condition you explained must hold to have the “average of average” being equal to the global average.
My point is based on that a parallel process just does the same thing in parallel as a serial process does, and both of them are supposed to produce identical results.
What I am wondering about is that the backward path of the computational graph in a DDP process starts from a local loss, while it starts from a global loss in the serial process, and they are supposed to produce the same result.
From your former explanation, I learned that the backward path starts from the global loss in DP, but not DDP. So, I believe that DP will produce the same results as the serial process does, but I wonder about DDP.
One thing I have come across is that, if the global loss is computed by sum() / batch_size, the backward path might start from 1 and dividing it by batch_size. If this is true, the only difference between starting from the global loss and the local loss should be difference between dividing by the global batch size and the local per-GPU batch size.
So, I suspect that the gradients in those cases have the same direction but different sizes. In particular, the gradient from DDP might be n_gpu times larger than DP, where n_gpu is the number of GPUs. Even if this is true, that will not be a big problem, but DDP may require a different learning rate from DP. I just thought that way, but it needs a confirmation.
Is this correct? I appreciate your assistance. Thank you.