marcelwa:

The tensors should stay part of the computation graph so that backpropagation works.
The question is why does Pytorch not free the memory in the GPU for tensors that are part of the computation graph?


PyTorch doesn’t free these tensors because they are needed for the backward pass to work properly, since they are part of the computation graph.
In your example you are creating new tensors x, w, and l, which doesn’t delete the computation graph. I.e. l.mean().backward() would still work.
You won’t be able to access the .grad attribute from x and w anymore, but the gradients will still be calculated. If you change the CPU tensors to e.g. x1 and w1, you would be able to print x.grad and w.grad after the backward pass.
You cannot move the computation graph to the CPU after its creation by moving leaf variables to the CPU (at least I’m not aware of a method to do so).