I think you would have to mask these parameters at some point (either directly or their gradients to prevent an update).
The optimizer cannot take parts of tensors, so I’m not aware of any other clean approach.