Solved by DAlolicorn in post #2 
              
              
                I think in your test, the forward loop does not involve gradient calculations. 
Thus the speed does not change much if no backward() is called, it may just save you some memory of the gradient placeholders.