I am trying to train a CNN distributedly. As the data-parallel nodes increases, the communication time gets longer. So I think overlap backward() and all_reduce() is a good way to reduce communication time. But I wonder how could I do this. Is there an easy way to do so?
My PyTorch version is 0.4.0.