You don’t need to detach the state. My last point was : taking Q as a loss, only the parameters given to the optimizer will be optimized.
The gradient update in DDPG is not that complicated. In fact, it’s even simpler than stochastic policy gradients:
Since Q approximates your returns, you want to maximize E_a~mu(s) [Q(a,s)] = E_s [Q(mu(s), s)]
So, you take the derivative of this thing : E_s [mu’(s) * Q’(mu(s), s)] and do a gradient descent on the parameters of mu.
With an automatic derivation tool like pytorch, you don’t even care about going that far, you just take -Q(mu(s), s) as a loss to maximize (wrt mu’s parameters).