Thanks @mrshenli for your reply. This is what MyModel do internally.
class MyModel(nn.Module):
    def __init__(self):
        self.sub_model1 = SomeModel1()
        self.sub_model2 = SomeModel2()

    def forward(self, input):
        out1 = self.sub_model2(input)
        out2 = self.sub_model1(out1)
        return out1, out2

I want the gradient to be synced eventually when I call backward() at line 11, so the above code seems correct? My goal is to accumulate gradient from both loss1 and loss2 to sub_model2 and accumulate gradient only from loss2 to sub_model1. That’s why I try to zero out grad in sub_model1.
Note that I use local_loss = loss1 + 0.0 * loss2 and local_loss = 0.0 * loss1 + loss2 to mask out part of the loss before calling local_loss.backward().
----------------update------------
class MyModel(nn.Module):
    def __init__(self):
        self.sub_model1 = SomeModel1()
        self.sub_model2 = SomeModel2()
        self.sub_model3 = SomeModel3()

    def forward(self, input):
        out1 = self.sub_model2(self.sub_model1(input))
        out2 = self.sub_model3(out1)
        return out1, out2

My goal is to accumulate gradient from both loss1 and loss2 to sub_model1 and accumulate gradient only from loss2 to sub_model2 and sub_model3. That’s why I try to zero out grad in sub_model2 and sub_model3.