Are you using batch norm layers in the model?
Note that during training, the running estimates will be updated, so even if you properly zero out the gradients and “remove the backward” pass, the output will still change in the next forward pass with the same inputs.
Could you try to set your model to .eval() to disable this effect for the sake of debugging?