Andrea_Rosasco:

If I don’t use retain_graph=True when calling backward to update the learning rate I get an error on the second iteration.


Is that on the ds_loss backward or acc_loss backward? Because the acc_loss will backprop in the same graph as the ds_loss so I think the retain_graph is required here.
For the slowdown, I guess it depends how many times you unroll your loop as you differentiate through all these iterations.