iffiX:

advantages_minib = advantages_minib.detach()


It worked for the first mini batch, as both backwards were made succesfully but when I backwarded with the second mini batch, I got the same error for the critic backward function. So I added
critic_loss.backward(retain_graph=True)

actor_loss.backward(retain_graph=True)

But got this error after the second pass:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [64, 1]], which is output 0 of TBackward, is at version 4; expected version 3 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).