Hey @Yi_Zhang, we are working on adding P2P to NCCL ProcessGroup backend. We just bumped up the NCCL submodule version in https://github.com/pytorch/pytorch/pull/41608 45.
For now, to work around it, you can create a sub group of 2 ranks, and  then use dist.broadcast(tensor, src, group=sub_group) to mimic P2P send/recv. PipeDream is already using that.
If you need general P2P support, you could try the RPC API 25. A caveat is that we are still working on improving support for GPU tensors. https://github.com/pytorch/pytorch/issues/41369 22