As far as I know, when in DDP(DistributedDataParallel), loss.backward()will synchronize gradient for all nodes in the group automatically through Reducer. However, If I do want to synchronize and update model parameters among part of nodes in some epochs, how can I manage to do that?
I would appreciate you for any hints or concrete code sample