marcelwa:

In my “production” code .cpu() does not free up memory as it does in this example


This is expected, as moving l to the CPU won’t detach it from the computation graph.
Add
l = l.detach()
print("requires_grad=True: l = l.cpu() {}". format(active_bytes()))

after the cpu() operation, the graph should be freed, and you should get 0 active bytes again.
Note that cuda(), cpu() and to() operations are differentiable, so that the computation graph will be kept alive and you can backpropagate through these operations.