That depends on your optimizer really. Chaging these parameters and statistics is the “right thing to do” if you really want to apply the optimizer.
If you want to freeze some weights manually, the right way to do it is not to give it to the optimizer or you can try and set weight.grad = None.