Solved by albanD in post #2 
              
              
                Hi, 
In pytorch, the weights are only updated when you do optimizer.step() which happens after the .backward() pass is completed. So no. 
In general, if you rewrite all your formulas to add a t parameter to your weights to differentiate the version before the update and the one after. You will see t…