Ok I see, it makes sens to directly derivates Q(S, pi(S)) wrt pi 's parameters.
In the paper, the gradient is wrt a, because they decompose the derivative (hence the mu_grad in my equation above). If you directly take Q as a loss, you must derivate it wrt policy’s parameters.
So, just doing
pi_loss = -Q(state, pi(state))
pi_loss.backward()
pi_optimizer.step()

should be ok.
In the paper, look at equation 6: it’s a mater of computing the first line (Q directly with derivative wrt pi) or the second line (decomposition, with a derivative wrt a)