Hi,
To prevent the autograd from saving anything, setting with torch.no_grad(): is enough.
The other issue you might have is about how you measure GPU memory as pytorch use a custom allocator and so memory reported by the OS is not necessarily correct.
Also when you do out = relu(out), the relu is first computed, and thus a temporary output is allocated. Then the old content of out is discarded and replaced with the temporary one. So I would expect to see a memory bump here.
Note that most activation functions have an inplace flag to make them perform their update inplace and avoid this issue.