Hello I am trying to incorporate multiple losses into a multi-model structure. My current workflow is as following.
optimizer = optim.SGD([{'params': model_1.parameters(), 'lr': 0.1},
                       {'params': model_2.parameters(), 'lr': 0.01},
                       {'params': criterion_1.parameters(), 'lr': 0.01}])

output_1, loss_1 = model_1(input)
output_2, loss_2 = model_2(output_1)
loss_3 = criterion_1(output_1, labels)
loss_4 = criterion_2(output_2, labels)

loss = loss_1 + loss_2  + loss_3 + loss_4

loss.backward()

optimizer.step()

I am worried if it will cause double gradient, because the backward() will go through loss_1, and loss_3, both, kinda, come from model_1. Similar as loss_2 and loss_4, while criterion_2 does not have parameters to be optimized.
I am also wondering what double gradient can cause if happened.
Thank you very much.