In the latest version at least, both optim.SGD and optim.Adam accept parameters with requires_grad=False. All parameters with requires_grad=False will have 0 gradients and so with plain SGD they won’t change. Note though that if there is l2 regularization or some momentum terms already saved for these parameters, they will change! Having requires_grad=False only means from the optimizer point of view that the gradients will be 0. The parameters might still be changed.