jwyao:

This problem can happen with other networks whose computational graph is random. To me, it is a good idea to make DDP handle this case without hard matching gradient updates from each worker. If a parameter doesn’t appear on one worker, the update for the parameter from this machine is just zero.


find_unused_parameters was supposed to handle this case. Which error did you see after setting find_unused_parameters=True?