Yes, correct, we currently don’t provide a way to do collectives on top of RPC/TensorPipe. The rationale is that the “native” collective libraries (NCCL, Gloo, MPI) are already doing a much better job at this, hence we’re not optimizing TensorPipe and RPC for that use case. However you should be able to combine RPC with the collective libraries very easily. Here is a tutorial showing how to do so with DDP, but if you prefer to use the “lower-level” API that should work too: Combining Distributed DataParallel with Distributed RPC Framework — PyTorch Tutorials 1.8.1+cu102 documentation