Hi. I have a question regarding data parallel (DP) and distributed data parallel (DDP).
I have read many articles about DP and understand that gradient is reduced automatically. However, I could not find an article explaining whether or not loss is also reduced. For example, I believe that the following codes appear typical main routine of a DP program.
outputs = model(inputs)
loss = criterion(outputs, targets)
loss.backward()

I understand that the split inputs and the model are copied on each GPU and a forward path is concurrently computed to yield the loss, then a backward path is also concurrently computed and finally all gradients are reduced to one.
Is the loss obtained by above code averaged over all the GPUs, which is exactly same as a loss computed by a serial program? Or, is the loss a value from just one GPU (gpu0)? I need to plot a loss chart, so I wonder if the loss is averaged over the GPUs.
The same question applies to outputs. I also need to compute training accuracy using outputs in above code. Does it hold the results of all the GPUs? If so, in what structure of a tensor are they stored?
Regarding DDP, above codes are written in each process running on respective GPU. In this case, how can I access the values on all the GPUs to plot the averaged loss and total accuracy?
I appreciate any sources of information. Thank you in advance.