I fix my problem like the FP16Optimizer in mmdetection 14, which is similar to the Apex delay all reduce.



 mrshenli:

One work around is to run forward-backward on the local model, and then manually run all_reduce to synchronize gradients after the backward pass.


And I realize the solution is just you said before… Thank you very much!!!
But it seems work no matter what value of find_unused_parameters. I wonder why it work when find_unused_parameters=True, the traverse should failed logically.