Hi albanD,
I think I’m running into a very similar problem.
I’m working on a Policy Gradient, in this algorithm you typically use some memory you sample from. The memory contains the last lets say 10 episodes the algorithm played. What you store are the log probabilities for the actions it chose. In this case you actually reuse data points even though they have already been updated, knowing that they are slightly wrong in the gradient, since they model would predict a slightly different probability after the first update. To be mathematically correct you would need to use importance sampling (p_old/p_new) but since this fraction is close to 1 of you use a sufficiently small learning rate this is typically dropped.
So some pseudo code would roughly look like this:
for episode in range(episodes_to_play):
    episode = play_episode()
    store_episode_in_memory(episode)
    drop_some_really_old_stuff_from_memory()
    log_probability, rewards = sample_some_memory()
    loss = REINFORCELOSS(log_probability, rewards)
    loss.backward(retain_graph=True)


I might have understood your explanation wrongly, but as far as I got it this is not possible anymore? The memory contains episodes that have already been used to train, but are still “fresh” enough to be not too far from the real distribution.
So my two questions:

Do I understand your explanation correctly and is the multiple iteration over the loss.backward() the problem why this throws errors, even if I use retain_graph=True?
How can I fix this and get back to the old behavior?

Thank you very much for your time!