Hi,
I have a network like this:
class Net(nn.Module):
    def __init__(self, subnet):
        ...
        self.subnet = subnet

    def forward(self,input):
        ...
        x = self.subnet(input)
        optimizer_of_subnet.zero_grad()
        loss = x.mean()
        loss.backward()
        optimizer_of_subnet.step()
        ...

If I run such code in multiple GPUs by DataParallel. Are the backward()s and optimizer_of_subnets in different GPU independent? Will they influence each other?
For example, backward() will be called four times if the code runs on 4 GPUs. If they are not independent, I might get 4 times larger gradients.
I think they are independent, but not so sure. I will appreciate any reply.