amirhf:

is the loss that is recorded later for only one process?


Hey @amirhf, if you are using DistributedDataParallel, yep, this is the local loss within one process. And different process can have different loss values.

Is summing and averaging all losses across all processes using ReduceOp.SUM  a better alternative?

This will give you the global loss, but will also introduce one more communication per iteration. If this is just for logging purpose, will it be sufficient if the logging is done every n iterations, so that there will be a smaller amortized comm overhead?