I would say it won’t affect for most of cases, unless you are working with techniques which involves gradients (like gradient alignment or so).
Can you check the ordering you are calling optimizers steps and zero grads? I think that’s probably the issue.