Solved by wayi in post #3 
              
              
                If you want to explore model parallelism in a distributed environment, you need to use Distributed RPC framework. 
The tutorial page of DDP + RPC can be found here: 
https://pytorch.org/tutorials/advanced/rpc_ddp_tutorial.html#