mrshenli:

The reason find_unused_parameters=True does not work is because, DDP will try to traverse the autograd graph from output at the end of the forward pass when find_unused_parameters is set to True. However, with checkpoint, the autograd graphs are reconstructed during the backward pass, and hence it is not available when DDP tries to traverse it, which will make DDP think those unreachable parameters are not used in the forward pass (although they are just hidden by checkpoint). When setting find_unused_parameters=False , DDP will skip the traverse, and expect that all parameters are used and autograd engine will compute grad for each parameter exactly once .


If so, then the new problem is caused by my codes. I will try to figure it out .
I understand the effect of find_unused_paramerts  now. Therefore, even my network has some unused parameters, but in each forward, each GPU updates the same unused parameters. In such a situation, I also can set  find_unused_paramerts=False all right ?