Your code is very long and I didn’t had time to look through it sorry.
But a question I would have is why do you set retain_graph=True? You don’t actually call backward on that same loss again so that shouldn’t be needed.