Solved by ptrblck in post #2 
              
              
                Yes, these tensors should be freed after the backward(). 
To double check it, you could use this example and add some print statements to check the memory: 
for t in range(5):
    # To apply our Function, we use Function.apply method. We alias this as 'relu'.
    relu = MyReLU.apply

    # Forward p…