Thank you for your reply!
Leaving out the retain_graph = True  in the backward() function does not have any effect on the training speed, nor on the performance. As you asked, I also looked at the memory usage. The processor usage is slightly bigger when adding the retain_graph = True. However, it does not change the training speed significantly.