In other words, even though discriminator’s optimizer won’t be called for the generator updates, you will still save some time and memory by turning requires_grad off. This is because regardless of whether or not there will be optimizer call on weight, weight.grad get calculated (if requires_grad is on).