Hi @mrshenli,
To follow up with the issue below, I wonder if you could let me know more about why currently we cannot use the “nccl” as a backend for the communication for RCP? We have to explicitly copy the data to CPU and then do the transmission.
What are the concerns that the RPC does not do something similar to the DDP that has the GPU directly access to the NIC for the model parallel algorithm?
Thank you



 mrshenli:

Hey @ZiyiZhu Are you trying to run this with RPC? Currently init_rpc does not work together with init_process_group . There are work around to create non-default process groups. Or we can also add a fix to init_rpc if necessary. This is the tracking issue: