oliver_ss:

To handle this problem, simply set the “find_unused_parameters=True” seems not to work, as least in my case.


Yep, this is true. Because that mode in DDP would require full access to all outputs and then traverse the graph from those outputs to find unused parameters. That’s also the reason that the error message says: “(2) making sure all forward function outputs participate in calculating loss.”

A simple solution I could find is adding a safety loss, this is, fetch all the outputs of my model and multiply them by 0. This would give all the parameters a 0 gradient but it might waster some cuda memory.

Another option might be only returning outputs that participate in computing loss. Other outputs can be stored in some model attributes, and retrieve them separately after the forward pass.