Thanks for the reply! Turns out I don’t need loss.required_grad = True and the .detach() call on the advantage function for the loss function to have grad enabled.
But now I receive the error message:

RuntimeError: Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward.

Seems like backpropagation goes through my network two times?
I’ve also tried something else, removing the advantage function (which is ofc not correct for actor-critic). Then for some reason the loss function does not have grad_enabled, and using grad_enabled = True results in the gradient of my weights being zero again.
Is_Leaf is true again though.
I think here is already something wrong.