To follow up with the issue below, I wonder if you could let me know more about why currently we cannot use the “nccl” as a backend for the communication for RCP?

This is because NCCL does not support p2p (send/recv) communication yet when we develop RPC. It is possible to use NCCL broadcast to mimic that send/recv, but that’s too hackish.
The p2p comm is coming to NCCL in v2.7. When that is ready, we probably can add it to ProcessGroupAgent or the new TensorPipeAgent (the latter is a more performant RPC agent implementation and should be able to use the best channels, e.g., IB/ETH/NvLink/etc.). See this PR: https://github.com/pytorch/pytorch/pull/35483 2

We have to explicitly copy the data to CPU and then do the transmission.

For Gloo backend even if application don’t copy the tensor from CUDA to CPU, Gloo would need to do that internally anyway. Hence, this explicit copy in application not a perf limitation when using Gloo backend.
We used to do that GPU-to-CPU copy implicitly in v1.4, but later realized that applications could run into unexpected errors if the destination device is not available on the callee. E.g., when I do rpc.rpc_sync(...., args=(torch.zeros(2).to(3),)) and if cuda:3 is not available on the callee, it would throw an error. So, we decided to make it explicit for applications.

What are the concerns that the RPC does not do something similar to the DDP that has the GPU directly access to the NIC for the model parallel algorithm?

From the API level, the difference is that DDP is supposed to run on a set of homogeneous servers, and RPC should be able to support heterogeneous clusters. So the device mismatch in RPC can be common. We are adding explicit device placement support (sth. similar to map_location on torch.save and torch.load) to the RPC API. This is an early issue to track. @osalpekar is working on a design RFC for that. Look forward to hear your comments when that RFC is posted.