I want to modify gradients manually with function tensor.register_hook().
I’ve tested that if training on one GPU, gradients will be collected correctly.
But I’m not sure if use DDP training, will the gradients also be collected correctly.
def modify_grad(self, g):
    return g * 3

feature.register_hook(self.modify_grad)

Say, if I changed the gradient on each GPU by multiplying 3, so 3 times of the original reduced gradient will be broadcast to each GPU?