The parameters with requires_grad = False won’t accumulate a gradient during the backward pass and their .grad attribute will stay None by default (if no previous gradient was calculated).
The optimizer will skip all parameters with a None gradient as seen here 16.


All parameters will accumulate gradients and the optimizer will only update the passed parameters. If you call optimizer.zero_grad() and don’t use model.zero_grad(), the “unused” parameters will continue to accumulate gradients. If you later decide to use these parameters in another optimizer, you should make sure to zero out their gradients before.