I got this error message when using DDP with 2 trainers on 2 machines. The training process worked for a few batches and then returned the following message. I was not able to fully figure out what happened in the training process. My understanding is: one worker didn’t return the gradient of some parameters while the other worker started the next iteration. That is, the gradient updates were not synchronized correctly.
I tried to add all training parameters outputs to the loss function and re-run the distributed training. But still received the same error message.
Could anyone help me figure out where the potential problem is?
Note: The problem goes away when worker number = 1.
 File "/home/tiger/usr_name/simgnn-dgl-torch/train_ray.py", line 117, in train_epoch
    adv_out, feature_out, feature_loss = model(block)
  File "/home/tiger/anaconda3/envs/usr_name/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/tiger/anaconda3/envs/usr_name/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 787, in forward
    if torch.is_grad_enabled() and self.reducer._rebuild_buckets():
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by 
making sure all `forward` function outputs participate in calculating loss. 
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameters which did not receive grad for rank 0: _adv_encoder._ff.ln.2.beta, _adv_encoder._ff.ln.2.alpha, _adv_encoder._ff.layers.0.weight, _adv_encoder._ff.layers.0.bias, _adv_encoder._ff.layers.1.weight, _adv_encoder._ff.layers.1.bias, _adv_encoder._ff.layers.2.weight, _adv_encoder._ff.layers.2.bias, _adv_encoder._ff.ln.0.alpha, _adv_encoder._ff.ln.0.beta, _adv_encoder._ff.ln.1.alpha, _adv_encoder._ff.ln.1.beta
Parameter indices which did not receive grad for rank 0: 11 12 13 14 15 16 19 20 21 22 23 24