Hi,
similar topic to this question 19: do optimizers work transparently in multiprocess runs or do I need to average the gradients of each process manually?
The imagenet example 6 in the pytorch/examples repo does not do explicit gradient averaging between processes, but the example on distributed 21 training in pytorch’s tutorials does.
Thanks a lot!
Enrico