Thanks for the answer and the tip 



 ptrblck:

Phase3 doesn’t use any parameters of netA to calculate the loss l3


This is the key problem.
In phase 2, netB is trained as a function of netA, so I want netA to be a part of the computation graph created by netB. So when netB giving predictions, netA will get gradients.
In other words,



 otb:

l2.backward(retain_graph=True)
netB_opt.step()


Calculate the gradients of netB w.r.t netA predictions and update netB, but it does not save this update as a function of netA (which it is what I want to do…)