the second piece of code is related to neural networks train phase and it works. while if we follow the same reasoning that you explained the second code shouldn’t work too. because it is in a for loop and at the end of loop the gradient history for both w1, and w2 are cleared while in next iteration they has been used in the loss.backward().