So the most likely problem is that you perform some operations in a differentiable manner outside of your training loop. And so these operations are part of the computational graph for multiple iterations.
Could you give a small (30/40 lines) that I can use to reproduce this?