Okay I figured out how to do this. And it was amazingly simple too… but I’ll leave this here anyway for anyone else in future who might stumble on the same problem because I had a really hard time googling for a solution.
act = actor(state)
act.retain_grad()
q = critic(act)
loss = -q
loss.backward()
new_act = act + act.grad * 0.01
q = critic(new_act)