No gradients are being calculated, but the intermediate activations are stored, which are necessary to calculate the gradients. If you wrap your code in this statement, the intermediates will be cleared, which also means you cannot call .backward() on your loss anymore.