It depends.
If they are needed to compute gradients, yes the autograd will keep them in memory.
If they are not needed to compute gradients, they will be freed when you associate a new Tensor to the python variable.