@ptrblck What about if modelC = modelB and I want to backpropagate through modelB, but optimize modelA?
modelA = nn.Linear(10, 10)
modelB = nn.Linear(10, 10)

# a is needed for both losses
x = torch.randn(1, 10)
a = modelA(x)

b1 = modelB(a.detach())
loss1 = loss_function_1(b1)
loss1.mean().backward()
# Here call optimForB.step()

b2 = modelB(a)
loss2 = loss_function_2(b2)
loss2.mean().backward()
# Here call optimForA.step()

I will need to compute
b1 = modelB(a.detach())

for the first loss and
b2 = modelB(a)

for the second. This seems kinda wasteful imo. Is there a way to detach after calculation, so I only have to calculate b = modelB(a) once?