Answers.Body: <p>This is not quite the loss that is stated in the paper.</p>
  <p>For standard policy gradient methods the objective is to maximise <span class="math-container">$v_{\pi_\theta}(s_0)$</span> -- note that this is analogous to minimising <span class="math-container">$-v_{\pi_\theta}(s_0)$</span>. This is for a stochastic policy. In DDPG the policy is now assumed to be deterministic.</p>
  <p>In general, we can write
  <span class="math-container">$$v_\pi(s) = \mathbb{E}_{a\sim\pi}[Q(s,a)]\;;$$</span>
  to see this note that
  <span class="math-container">$$Q(s,a) = \mathbb{E}[G_t | S_t = s, A_t=a]\;;$$</span>
  so if we took expectation over this with respect to the distribution of <span class="math-container">$a$</span> we would get
  <span class="math-container">$$\mathbb{E}_{a\sim\pi}[\mathbb{E}[G_t|S_t=s, A_t=a]] = \mathbb{E}[G_t|S_t=s] = v_\pi(s)\;.$$</span></p>
  <p>However, if our policy is deterministic then <span class="math-container">$\pi(\cdot|s)$</span> is a point mass (a distribution which has probability 1 for a specific point and 0 everywhere else) for a certain action, so <span class="math-container">$\mathbb{E}_{a\sim\pi}[ Q(s,a)] = Q(s,a=\pi(s)) = v_\pi(s)$</span>. Thus the objective is still to maximise <span class="math-container">$v_\pi(s)$</span> it is just that now we know the policy is deterministic we say we want to maximise <span class="math-container">$Q(s,a=\pi(s))$</span>.</p>
  <p>The policy gradient of this term was shown to be
  <span class="math-container">\begin{align}
  \nabla_\theta Q(s,a=\pi_\theta(s)) &amp; \approx \mathbb{E}_{s \sim \mu}[\nabla_\theta Q(s,a=\pi_\theta(s))]\;; \\ &amp; =  \mathbb{E}_{s\sim\mu}[\nabla_aQ(s,a=\pi(s)) \nabla_\theta \pi_\theta(s)]\;;
  \end{align}</span></p>
  <p>where if we put a minus at the front of this term then we would arrive at the loss from the paper. Intuitively this makes sense, you want to know how much the action-value function changes with respect to the parameter of the policy, but this would be difficult to directly calculate so you use the chain rule to see how much the action-value function changes with <span class="math-container">$a$</span> and in turn how much <span class="math-container">$a$</span> (i.e. our policy) changes with the parameter of the policy.</p>
  <p>I realise I have changed notation from the paper you are reading so here <span class="math-container">$\pi$</span> is our policy as opposed to <span class="math-container">$\mu$</span> and here where I have used <span class="math-container">$\mu$</span> I take this to be the state distribution function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-22T12:49:17.607Z
Answers.LastActivityDate: 2023-06-20T17:02:57.857Z
Answers.LastEditDate: 2023-06-20T17:02:57.857Z
Answers.UserDisplayName: David
Body: <p>I am trying to implement the DDPG algorithm based on this <a href="https://arxiv.org/abs/1509.02971" rel="nofollow noreferrer">paper</a>.</p>
  <p>The part that confuses me is the actor network's update.
  I don't understand why the policy loss is simply the mean of <span class="math-container">$-Q(s, \mu(s))$</span>, where <span class="math-container">$Q$</span> is the critic network and <span class="math-container">$\mu$</span> is the policy network.
  How does one arrive at this?</p>
  
Tags: <reinforcement-learning><policy-gradients><pytorch><ddpg>
Title: Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?
UserDisplayName: Dhanush Giriyan
ThreadText: # Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?
  
  (Asked by: username_0 on 2020-07-22)
  
  I am trying to implement the DDPG algorithm based on this [paper](https://arxiv.org/abs/1509.02971).
  
  The part that confuses me is the actor network's update.
  I don't understand why the policy loss is simply the mean of $-Q(s, \mu(s))$, where $Q$ is the critic network and $\mu$ is the policy network.
  How does one arrive at this?
  
  **Tags:** reinforcement-learning, policy-gradients, pytorch, ddpg
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-22. Score: 3 | Accepted answer
  
  This is not quite the loss that is stated in the paper.
  
  For standard policy gradient methods the objective is to maximise $v\_{\pi\_\theta}(s\_0)$ -- note that this is analogous to minimising $-v\_{\pi\_\theta}(s\_0)$. This is for a stochastic policy. In DDPG the policy is now assumed to be deterministic.
  
  In general, we can write
  $$v\_\pi(s) = \mathbb{E}\_{a\sim\pi}[Q(s,a)]\;;$$
  to see this note that
  $$Q(s,a) = \mathbb{E}[G\_t | S\_t = s, A\_t=a]\;;$$
  so if we took expectation over this with respect to the distribution of $a$ we would get
  $$\mathbb{E}\_{a\sim\pi}[\mathbb{E}[G\_t|S\_t=s, A\_t=a]] = \mathbb{E}[G\_t|S\_t=s] = v\_\pi(s)\;.$$
  
  However, if our policy is deterministic then $\pi(\cdot|s)$ is a point mass (a distribution which has probability 1 for a specific point and 0 everywhere else) for a certain action, so $\mathbb{E}\_{a\sim\pi}[ Q(s,a)] = Q(s,a=\pi(s)) = v\_\pi(s)$. Thus the objective is still to maximise $v\_\pi(s)$ it is just that now we know the policy is deterministic we say we want to maximise $Q(s,a=\pi(s))$.
  
  The policy gradient of this term was shown to be
  \begin{align}
  \nabla\_\theta Q(s,a=\pi\_\theta(s)) & \approx \mathbb{E}\_{s \sim \mu}[\nabla\_\theta Q(s,a=\pi\_\theta(s))]\;; \\ & = \mathbb{E}\_{s\sim\mu}[\nabla\_aQ(s,a=\pi(s)) \nabla\_\theta \pi\_\theta(s)]\;;
  \end{align}
  
  where if we put a minus at the front of this term then we would arrive at the loss from the paper. Intuitively this makes sense, you want to know how much the action-value function changes with respect to the parameter of the policy, but this would be difficult to directly calculate so you use the chain rule to see how much the action-value function changes with $a$ and in turn how much $a$ (i.e. our policy) changes with the parameter of the policy.
  
  I realise I have changed notation from the paper you are reading so here $\pi$ is our policy as opposed to $\mu$ and here where I have used $\mu$ I take this to be the state distribution function.
  
  
  ---
  
  