Solved by ptrblck in post #4 
              
              
                I think you are right and you should see the expected behavior, if you use an optimizer without internal states. 
Currently you are using Adam, which stores some running estimates after the first step() call, which takes some memory. 
I would also recommend to use the PyTorch methods to check the al…