If I understood it correctly, when I would detach the tensor, then backward() does not work anymore and then there would no point of using Pytorch anymore.
Additionally, changing the second part to
# requires_grad=True
x = torch.randn((256, 128), device='cuda', requires_grad=True)
w = torch.randn((128, 512), device='cuda', requires_grad=True)
l = torch.matmul(x, w)
print("requires_grad=True {}". format(active_bytes()))

x = x.cpu()
print("requires_grad=True: x = x.cpu() {}". format(active_bytes()))
w = w.cpu()
print("requires_grad=True: w = w.cpu() {}". format(active_bytes()))
l = l.cpu()
print("requires_grad=True: l = l.cpu() {}". format(active_bytes()))

with the output
requires_grad=True 917504
requires_grad=True: x = x.cpu() 917504
requires_grad=True: w = w.cpu() 917504
requires_grad=True: l = l.cpu() 393216

shows that the inputs to the matrix multiplication are not freed.
The tensors should stay part of the computation graph so that backpropagation works.
The question is why does Pytorch not free the memory in the GPU for tensors that are part of the computation graph?
The tensors are not lost, they are just stored somewhere else.