I initially thought this would solve the problem but unfortunately it didn’t. I will leave the comment though.
I think I have also verified that this happens due to this particular change, using PyTorch 1.4 it works perfectly fine

I think I have found a solution, assuming that my understanding of the problem is accurate:
Simply cloning the loss before calling .backward(). Is that an accurate solution or did I miss something?
for episode in range(episodes_to_play):
    episode = play_episode()
    store_episode_in_memory(episode)
    drop_some_really_old_stuff_from_memory()
    log_probability, rewards = sample_some_memory()
    loss = REINFORCELOSS(log_probability, rewards).clone()
    loss.backward(retain_graph=True)