Python uses function scoping, which frees all variables which are only used in the function scope.
Your memory footprint should therefore be lower, as e.g. out will be deleted and with it the intermediate tensors, which were created in the forward method. My code snippet doesn’t use it and thus the “first” out tensor with the computation graph is still in memory while the second iteration is running.
In your second example, the memory might just be reused. I’m not sure how the CPU memory allocation works in Python and PyTorch in particular. For GPU memory we use a custom caching allocator, which reuses memory if possible without reallocating.