Description of the problem
Hi,
I tried to train the model with DDP and a strange question occurred to me.
The training process worked really well if DataParallel was used instead of DistributedDataParallel.
When I wrapped the model with DDP,  the exception below was raised:
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_para
meters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If
 you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return va
lue of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module whe
n reporting this issue (e.g. list, dict, iterable).



I have tried to set “find_unused_parameters=True”, but the training process would get stuck. Specifically, the loss failed to go backward.


I have also checked all the parameters in my model with the method proposed here: How to find the unused parameters in network 79
I am really sure all the parameters have changed after several iterations, but I would still meet the “RuntimeError” as above.


Ps: The output of the model is a list of tensors.
I am really confused and wondering whether there exists any tips to help solve this problem.
Thank you!