There is an “easy” way of doing this. By actually specifying parameter-specific learning rates and setting them to 0 as already suggested in the other comments.
As you already found out you can’t specify each specific parameter in the optimizer like so: {'params': model.conv.weight[0, 0, 0, 0], 'lr': 0}. This raises the ValueError: can't optimize a non-leaf Tensor.
But you can specify a tensor for the learning rate that has the same shape as the parameter group: {'params': model.conv.weight, 'lr': tensor.zeros_like(model.conv.weight)}. If you do so you will encounter another problem: The optimizer e.g. Adam doesn’t allow a tensor as learning rate as addcdiv_() expects a scalar for its argument value. Therefore you need to change this last line in the step method to: p.data.add_((-step_size * (exp_avg / denom)))