Yes, if some parameters and thus operations don’t need gradients, the intermediate tensors won’t be stored. The same would happen, if you wrap the code in a with torch.no_grad() block to save memory during the validation pass.