It should be possible. And there are several levels of APIs that you can use:

send/recv APIs: https://pytorch.org/docs/stable/distributed.html#torch.distributed.send 2

collective communication APIs: https://pytorch.org/docs/stable/distributed.html#torch.distributed.all_reduce

RPC APIs:
a. https://pytorch.org/docs/stable/rpc.html 1
b. https://pytorch.org/tutorials/intermediate/rpc_tutorial.html 2
c. https://pytorch.org/tutorials/intermediate/rpc_param_server_tutorial.html 1