Solved by Joshua_Clancy in post #2 
              
              
                Ahhhhh finally figured it out. For anyone else with a similar problem, it was because I had retain_graph = True on the first backward function. In other contexts that is useful with multiple losses/models but not in this instance.