tom:


with keep_graph=True (or keep_variables=True in pytorch <=0.1.12) in the first call, you will do the same as in 3 and five: You backprop twice to compute derivatives at the last evaluated point. The .grads are added to the .grads you had, so you end up computing twice the gradient at the last evaluated point.
without keep_graph=True in the first call, pytorch can throw away the processed bit’s information from forward, and will give an error meaning “forward info is gone, you used it and didn’t tell me to keep it”.



Pointing out that keep_graph is now called retain_graph.