No, you should not add “retain_graph=True” because in PPO actor does not need gradient from critic.

So whenever we call  retain_graph=True  it applies to the next backward call, no matter to which network we applied it initially?
Thank you so much for the help! It runs perfectly right now!
I would really appreciate it if you have time to explain to me why returns_traj[minibatch_ind].detach() and advantages_minib = advantages_minib.detach() played so important role in this error.