No, you should not add “retain_graph=True” because in PPO actor does not need gradient from critic.
you probably should also detach:
returns_traj[minibatch_ind]

basically you actor gets gradients through entropy and your critic gets gradients from value loss. Anything other than these two tensors should also be detached.