You can use point-to-point communication 5 or collective operations 4.
For example, a tensor could be sent by torch.distributed.send and received by torch.distributed.recv functions by specifying the target ranks.
You can choose to broadcast or reduce if you wish. I usually use torch.disbtibuted.all_reduce function to collect loss information between processes.
Example here 3.
If you use nccl backend, you can only use CudaTensor for communication.