How is Multiple node, Multiple worker Allreduce implemented in PyTorch?
I know that in a single node multi-worker setting, allreduce is implemented with a ring allreduce algorithm. How does this change in a multinode setting?