Thanks so much for the comments! I did get an error without retain_graph=True, I can try removing that and checking again.
Good catch on the optimizer being reinitialized every epoch.
I changed the code to not use Variables any more, thanks for that.
I realized the loss was being accumulated which is a dumb mistake. Have corrected that. Will try running and see how the code performs.