OK, that makes sense.
The memory usage won’t stay the same, since for each pass a new computation graph is created and stored.
You could call .backward() in the for loop and optimizer.step() outside of it.