So I will not insist on this beyond further in this thread, but my exact observation is that

otherwise I have the typical trying to backward through the graph a second time error.

is generally not a good reason to use retain_graph=True. Personally, I always try to articulate it very explicitly “I need to retain the graph for the calculation of FOO from BAR across training steps because of BAZ”. It seems silly, but more often than not, I’ve seen people discover they don’t need retain_graph=True after all. But as you were happy with it, I’ll leave it at this general comment.



 perceptron1:

The algorithm works fine without this new loss I am trying to add, but as soon as I add it I get the error, very strange.


So a bit of background: Autograd will save inputs and/or outputs of the forward in order to compute the backward, but it will only save those actually needed (modulo coding errors, which are unlikely in stock PyTorch functions).
This means that there are two ways of introducing a fresh trying to backward...error:

The perhaps more obvious one: Introduce a new inplace operation.
The perhaps more subtle one: Change the computation of something (or the computation with something) that is modified inplace from something that doesn’t need the output (or input, respectively) for computing the backward to something that does need it for computing the backward. So in this case, not the inplace is new, but how you compute what gets changed inplace (or what you compute with it).

Usually, the anomaly detection should give you a hint, in particular if it points to a new line of code, it could be the second case.
Best regards
Thomas