Dagenham:

So in other words after one step(), the next backward() call doesn’t make much sense because the loss was calculated using old parameters? Am I understanding it correctly?


Yes, that’s correct. Additionally to this the intermediate activations were also calculated using the “old” parameters. The backward pass would use the new parameters with the old intermediates to calculate the gradients for the new parameters, which would be wrong.



 Dagenham:

If so, one way to fix this error would be to execute forward passes in the loop as well, right?


Yes, that would be an alternative solution to your first approach (calling backward and step after the loop).



 Dagenham:

Sorry if my train of thought (or wording) is confusing.


Don’t worry, it’s not at all confusing.