Thanks!!!
According to my test, gradients on different GPUs don’t seem to affect each other.
But I found another strange phenomenon: the backward hooks registered in tensors of subnet are missed(not called) in some cases. I check this by following code:
class Subnet(nn.Module):
    def __init__(self):
        self.hook_flags = [0,0,0,0] #run on 4 GPUs
    ...
    def forward(self, input):
        device_id = input.device.index
        assert(self.hook_flags[device_id] == 0) #flag decrease by 1 in hook, increase by 1 in forward, so it should be 0 here
        self.hook_flags[device_id] += 1           
        x = self.layer1(input)
        ...
        x.register_hook(lambda grad: self.hook(grad, device_id))
    def hook(self,grad,device_id):
        self.hook_flags[device_id] -= 1

The assertion fails occasionally, so I think hooks are missed in some cases. I can’t understand. Is my code wrong？Or hooks are really missed? I will appreciate your help.
Note that  backward() in forward() is called 4 times (4 GPUs) as my first post showed.