That seems to be the case, check out how they do it in the implementation of Mask-RCNN 101. They use reduce instead of all_reduce because, for logging, you only need the reduced and averaged values on the rank 0 process.