Hi @mrshenli, thanks for reply. I was following this example to use a average_gradients function, which calls all_reduce:

So to disable all_reduce, I just didn’t call this function when doing the training. And I notice there are two different examples, the other one uses DDP model and the sync-up step is done in backward computation, but I want to have a customized sync-up method, that’s why I choose the way with a separate average_gradients function.
I have tried using SimpleQueue to pass some large data, but it seems slow when calling .get(), I’ll try to use it for passing grad tensors and see how it performs.