Depending on the used optimizer, parameters might be changes even with a zero gradient, if running estimates are used, which would be the case for e.g. Adam.
Which optimizer are you using and are you zeroing out the gradients manually after the backward pass?