There are two options:

A higher-level API RemoteModule (recommended):
Distributed RPC Framework — PyTorch master documentation 16


You don’t need to explicitly write any RPC. Instead, you need to override the forward method. When you construct this nn.Module like module, need to explicitly specify the device placement (on a remote GPU in your case). The input tensors will be automatically placed to the same cuda device.
Another example can be found here: Combining Distributed DataParallel with Distributed RPC Framework — PyTorch Tutorials 1.8.1+cu102 documentation 5

A lower-level API RRef:
Getting Started with Distributed RPC Framework — PyTorch Tutorials 1.8.1+cu102 documentation 3


You need to write your own RPC, and call to_here() to run your remote module in the RPC.