Solved by albanD in post #4 
              
              
                It depends. 
If they are needed to compute gradients, yes the autograd will keep them in memory. 
If they are not needed to compute gradients, they will be freed when you associate a new Tensor to the python variable.