Solved by ptrblck in post #8 
              
              
                PyTorch doesn’t free these tensors because they are needed for the backward pass to work properly, since they are part of the computation graph. 
In your example you are creating new tensors x, w, and l, which doesn’t delete the computation graph. I.e. l.mean().backward() would still work. 
You won…