Hey @xerxex, there is a torch.distributed.rpc package for this purpose. Please refer to the following docs:

API doc: https://pytorch.org/docs/master/rpc.html 123

Tutorial: https://pytorch.org/tutorials/intermediate/rpc_tutorial.html 143


For now, we do not yet support creating remote device like torch.device('worker1/cuda0'), but this is on our roadmap and we plan to implement this as a sugar layer on top of RPC. Applications should be able to do the same thing using our raw RPC API.