Your approach looks correct.
Note that Adam (and other optimizers with internal running estimates) could still change all parameters, even if their gradient is zero, which might create the issue you are seeing.