Solved by albanD in post #2 
              
              
                Hi, 
To prevent the autograd from saving anything, setting with torch.no_grad(): is enough. 
The other issue you might have is about how you measure GPU memory as pytorch use a custom allocator and so memory reported by the OS is not necessarily correct. 
Also when you do out = relu(out), the relu i…