Oh I see, I guess I was confused by the name retain_graph. I’ve searched a bit and see that it was called retain_variables before. So I guess if I use retain_graph=True while putting loss.backward() inside the for loop, it defeats the purpose of saving memory because it will keep in memory the temporary tensors needed for the previous gradient, right ?