Yes sorry I said that wrong. The graph is destroyed when you call .backward() using torch no grad may lower your gpu usage slightly but not a lot. I have tried both ways and for some reason my gpu memory has never really been different. You can keep it in however if it impacts your performance. I am still confused what y2 does though.