Small addition:
I would suggest to mask the gradients instead of the weights directly, so that they won’t be updated.
However, I think weight decay might still update these parameters even if the gradient is zero.