If you want to explore model parallelism in a distributed environment, you need to use Distributed RPC framework 15.
The tutorial page of DDP + RPC can be found here:
https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html# 21