Hello,
I am reviewing the pytorch imagenet 10 example in the repos and I have trouble comprehending the loss value that is returned by the criterion module. In Line 291 14, is the loss that is recorded later for only one process? Is summing and averaging all losses across all processes using ReduceOp.SUM a better alternative? For example, when I want to save my model or simply log the metric, I would like to do it based on the average loss values across all processes.
In other words, is losses.update(loss.item(), images.size(0)) only saving the loss value for one process?