Hi Thomas,
Sure, I understand the general concern for knowing when to do retain_grap=True. I agree with you. I skipped the explanation of why I need it because I think its irrelevant to the current problem (I’ve been working on the same code for a while, with satisfactory results, and just encountered this error when adding a new custom loss). But thanks for your emphasis on the retain_graph=True issue, I think your logic should always be applied.
I think my problem is more likely your second bullet point.
One question that could potentially be related to the problem I have:
The current setup I have works like a recurrent neural network. The same network is called T (say T=50) times and the output of the network at time t (canvas_t) will be the input (concatenated with the same target image for the whole sequence T) of the network at time t+1. Every t%k (k is a hyperparameter, say k=10), I call backward on the cumulative losses at each t and peform a optimization step. If I detach the gradient of the output canvas_t after feeding it into the same network in the following time step, how is the network still able to backpropagate the gradients?
Thanks,