A follow up question:
In the same setting, what if loss1 only updates network A, and loss2 only updates network B, i.e.
optim_B.zero_grad()
loss2.backward(retain_graph=True)
optim_B.step()

optim_A.zero_grad()
loss1.backward()
optim_A.step()

Can we save calculation time by prevent back-propagation from loss2 through A?
Thank you for your time!