Hi,
What are possible reasons for gradients to differ across GPUs after a backward() call, when using DistributedDataParallel (DDP)? If I understood correctly wrapping the model with DDP in the main worker should take care of gradient averaging and their synchronization across GPUs?
Thank you