Many thanks for your informative reply!
I first thought about approach 2 (zero out the gradient). However, I am not familiar with register_hook, so implement this approach before optimizer.step(), I manually zero out the gradients. In this case, if I init the optimizer with passing weight_decay argument (not zero), the optimizer still makes the parameter updated. (ref here 4)
Therefore, I think approach 1 (separate parts of parameter) might be better in practice. Explicitly tell the optimizer which parameters should be updated. (I think set requires_grad is also good, checked here 8)
Note: I am not sure whether register_hook can deal with the above mentioned update by weight_decay.