I would guess that you do operations that require gradients outsite of your training loop. And so this part is common to every batches and will be ok for the first one. But the second one will try to use this same part of the graph and will fail because you already backproped through it.