Solved by albanD in post #2 
              
              
                Hi, 
The second one is most likely a bug. 
Remember that .backward() accumulates gradients and so zero_grad() should be called such that you clear the old gradients before accumulating the new ones. 
The second example here actually zero out the gradients just before using them. So the step won’t do…