This issue might be related to your optimizer.
If you use an optimizer with a momentum or some averaged statistics and pass all parameters to it, your inactive parameters might still be updated.
Which optimizer are you currently using?
I think the safest way would be to create two (or even three) optimizers and pass the corresponding parameters to each of them.