LicharYuan:

It’s ok in find_unused_parameters=False , so I guess maybe manually define checkpoint can solve my problem?


The reason find_unused_parameters=True does not work is because, DDP will try to traverse the autograd graph from output at the end of the forward pass when find_unused_parameters is set to True. However, with checkpoint, the autograd graphs are reconstructed during the backward pass, and hence it is not available when DDP tries to traverse it, which will make DDP think those unreachable parameters are not used in the forward pass (although they are just hidden by checkpoint). When setting find_unused_parameters=False, DDP will skip the traverse, and expect that all parameters are used and autograd engine will compute grad for each parameter exactly once.

But it makes new problems (Not Enough Values to unpack in backward)…

Could you please elaborate more on this problem?