I Think it’s just a coincidence unless you have some bug , for example, you missed some parameter, you are wronly calling zero_grad and backward.
For example, in your code above you are only passing those parameters which are “trainable”, however gradients keep being accumulated as they won’t be affected by optim.zero_grad(). Parameters “after” those ones will be exposed to accumulated gradients. I doubt that’s intentional.