Solved by ptrblck in post #3 
              
              
                Intermediate tensors will be freed, once backward() (without retain_graph=True) is used and these intermediates are not needed for the gradient calculation anymore. 
If you don’t need to compute the gradients and call backward(), you can wrap the forward pass into a with torch.no_grad() block, which…