Thanks for the response, Thomas!
So with “my” setup you would basically call 2-times forward before you call backward(), and since backward clears the graph, you have a larger graph in the background in my case. I fact, now I think that both approaches I listed above may be equivalent (depending on how autograd is implemented) in terms of the number of operations performed by backward. In that case, the approach in the tutorial is probably preferred because you have a smaller graph, keeping memory consumption lower.
Btw when I benchmarked both approaches, both approaches had similar speed. But yeah, it’s a relatively small network and image size.