I think what you may be looking for is our Distributed RPC framework (https://pytorch.org/tutorials/intermediate/rpc_tutorial.html?highlight=rpc 6), which allows you to send messages and tensors between workers. Also see the Distributed Autograd Framework (https://pytorch.org/docs/master/rpc.html#distributed-autograd-framework 2) for training models that are partitioned across machines. Lastly, here is an example of training an RNN using RPC/Distributed Autograd: https://github.com/pytorch/examples/tree/master/distributed/rpc/rnn 2.