Thank you for the useful explanations.
From the discussion above I understand that the reason why one shouldn’t do an all_gather sum of the losses when training Distributed Data Parallel mode is that these all gather operations can slow down the process.
Are there any other reasons why the loss tensors should not be summed other than performance reasons?
I ask this because in case the loss tensors are small, if an all_gather sum is performed when computing the losses, this will result in identical losses for all processes.  Therefore gradient averaging over processes will simply divide the losses by the number of processes.
This has the advantage of mimicking the behavior of DataParallel and of providing consistent results independently of the number of processes being run without the need to adjust learning rates, batch sizes, etc.
In short, when the cost of doing an all_gather sum of the losses is low, are there any other reasons beyond performance not to do it?  And isn’t the consistent behavior independently of the number of processes an advantage?
Thank you