Thanks for the reply!



 wayi:


It will be much more efficient if you use DDP instead of implementing your own communication by allreduce primitive.
Try no_sync context manager on the tutorial, which can reduce the sync frequency by accumulating gradients locally.



Unfortunately I can’t currently wrap my models with DDP (I’m integrating with another library that doesn’t play very well with DDP), but my plan is to do so as soon as possible.
That said, the application I’m developing for requires inter-process communication in both the forwards and backwards passes. The total communication is quite a bit higher than just the setup communication / gradient synchronization performed by DDP, so I suspect that wrapping my model with DDP won’t do very much to reduce communication costs.



 wayi:


If you can use NCCL, it will be much faster. You can also try DDP communication hooks for gradient compression.



I’d like to use NCCL, but it doesn’t directly support some of the communication primitives I need. In any case, for now I’m less concerned about raw communication speed and more concerned with how it scales as the number of processes increases.
If I used the NCCL backend, would the communication speed decrease at the same rate as Gloo (so that e.g. doubling the number of processes / process groups causes all_reduce to run at half speed)?