@mrshenli Above code works with mp.spawn (locally) but not with mp.Process (distributed way) way (Even after moving options init to workers_init_rpc function). Reason I am not using mp.Spawn is, like I said in my previous post I have 3 nodes which has variable number of GPUs (1, 2 & 4). My model is divided into 2 parts. First part reside on a node which has 4 GPUs, and modules of this part are divided (s.t. each GPU holds nearly equal number of parameters) onto these 4 GPUs. Second part reside on the machine which has 2 GPUs, and it’s modules are divided onto 2 GPUs (just like the previous one). Unfortunately, I can’t fit the whole model on a GPU, all of these GPUs have 8 GB VRAM which is not enough for my case. That is why I am using mp.Process methodology to do the training.
Furthermore, node with 1 GPU acts as a master, it encompasses these 2 parts of the model into one. Calls them (using rpc_sync) sequentially to run a full forward pass. Also, this dividing work (copying modules to GPUs) is being done in __init__ (think of Net1 and Net2 as 2 parts of the model) methods, could this be a problem when using mp.spawn?
When using mp.Process master node is successfully able to fetch the results from workers by making rpc_sync calls. Problem comes when executing dist_autograd.backward(context_id, [losses]) on master node, it hangs on this line and due to that ETIMEDOUT error is generated (After moving options line to workers_init_rpc method). This is true for my case (with GPUs) & the code I posted here (to repro. this error).
Note:
PyTorch Version: 1.7.1 (In all 3 of them)
Forgot to mention a observation,



 matrix:

StackTrace:
Process Process-1:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pytorch2/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/user/anaconda3/envs/pytorch2/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "main.py", line 70, in workers_init_rpc
    rpc.shutdown()
  File "/home/user/anaconda3/envs/pytorch2/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 78, in wrapper
    return func(*args, **kwargs)
  File "/home/user/anaconda3/envs/pytorch2/lib/python3.7/site-packages/torch/distributed/rpc/api.py", line 284, in shutdown
    _get_current_rpc_agent().join()
RuntimeError: [/opt/conda/conda-bld/pytorch_1607370156314/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [192.168.13.205]:28380



Every time I execute this (on workers), a different port is used (its not constant). Below, port 28380 is used on a particular worker.



 matrix:

RuntimeError: [/opt/conda/conda-bld/pytorch_1607370156314/work/third_party/gloo/gloo/transport/tcp/pair.cc:575] Connection closed by peer [192.168.13.205]:28380


Is this okay?