@mrshenli I made some changes like you said, now my code works. However there’s a problem.
On master node I get following output as I have put prints in my code:
Batch forward complete
Epoch: 0, Batch Id: 0,Train Loss: 16.620447158813477
Before dist_autograd (Before executing dist_autograd.backward())
Step done (After optimizer.step() is executed, of course its an instance of DistributedOptimizer)
Batch forward complete (Completion of the forward pass of a batch)
Epoch: 0, Batch Id: 1,Train Loss: 12.148786544799805
Before dist_autograd

It gets stuck here. There’s no error messages on either of the nodes. Here rpc_timeout is set to 1 hr.
Then I changed rpc_timeout to 1 minute, and got the following on master node
Batch forward complete
Epoch: 0, Batch Id: 0,Train Loss: 58.77790832519531
Before dist_autograd
Step done
Batch forward complete
Epoch: 0, Batch Id: 1,Train Loss: 40.98801803588867
Before dist_autograd
Process Process-1:
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/pytorch2/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/user/anaconda3/envs/pytorch2/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "main.py", line 65, in workers_init_rpc
    run_master()
  File "main.py", line 49, in run_master
    train_one_epoch(model, opt, data_loader, epoch, print_freq=10)
  File "/home/user/Documents/Incremental_Learning/demo/helpers/engine.py", line 45, in train_one_epoch
    dist_autograd.backward(context_id, [losses])
RuntimeError: Error on Node 0: RPCErr:1:RPC ran for more than set timeout (60000 ms) and will now be marked with an error

and below on the workers,
[W tensorpipe_agent.cpp:545] RPC agent for worker1 encountered error when reading incoming request from worker0: ECONNRESET: connection reset by peer (this is expected to happen during shutdown)

It does timeout after 1 minute. There’s no problem when processing first batch, it’s smooth doesn’t take time. But when processing second batch dist_autograd hangs, as you can see in above output. This is strange behaviour.