Hi, I am trying to implement DDPG with pytorch, I am sure most of my implementation is right, but the policies don’t converge so I am not too certain about the gradient part. The paper shows that the gradient of Q(s,a) is with respect to action a. I am defining the loss to be Q(s,a), but how do I know that the gradient will be taken derivative with respect to a? Since the loss = Q(s,a), I am setting loss.requires_grad = True, and loss.volatile = False.