Answers.Body: <p>Actually, your result that the gradient is 0 is correct given your formulation. Indeed, that is why one might have believed that the deterministic policy gradient didn't exist.</p>
  <p>The term <span class="math-container">$\nabla_{\theta}\log \pi(a \, | \, s)$</span>
  is a type of gradient estimator known as a  <em>likelihood ratio</em>, and it assumes that the support of <span class="math-container">$\pi$</span> does not depend on <span class="math-container">$\theta$</span>. In other words there must be some non-zero probability of choosing all the possible actions <span class="math-container">$\mathcal{A}$</span>, and <span class="math-container">$\theta$</span> encodes those probabilities. In your idea, <span class="math-container">$\mathcal{A}$</span> is a <em>single action</em>, and therefore the parameters don't do anything at all. You always have probability 1 of choosing that action. We can't let different parameters choose different actions, as that violates the assumptions of the likelihood ratio and will lead to a biased gradient.</p>
  <p>But why can't we let the support of a likelihood ratio depend on <span class="math-container">$\theta$</span>? I won't prove this in a rigorous way here but you can consider
  <span class="math-container">\begin{align*} 
  \mathbb{E}_{y} [ \nabla_{\theta}\, \log p_{\theta}(y)] = \mathbb{E}_{y} \left[ \underset{\epsilon \rightarrow 0}{\lim} \dfrac{\log p_{\theta+\epsilon}(y) - \log p_{\theta}(y)}{\epsilon}\right]
  \end{align*}</span>
  and see that if <span class="math-container">$p_{\theta + \epsilon}(y) = 0$</span>, then <span class="math-container">$\log p_{\theta+\epsilon}(y) = -\infty$</span>.</p>
  <p>I think a more instructive example is to consider what can go wrong if we just go ahead and let the support depend on <span class="math-container">$\theta$</span> anyway. Let's consider <span class="math-container">$\nabla_{\theta} \, \mathbb{E} [u]$</span> where <span class="math-container">$u \sim \text{Uniform}(0, \theta)$</span> (so <span class="math-container">$\theta$</span> is a scalar, and <span class="math-container">$u$</span> has support on <span class="math-container">$[0, \theta ]$</span>). We know <span class="math-container">$\mathbb{E} [u] = \frac{1}{2}\theta$</span> so <span class="math-container">$\nabla_{\theta}\mathbb{E} [u] = \frac{1}{2}$</span>. You should work out for yourself that the gradient estimator (using a likelihood ratio) gives the utterly wrong result of <span class="math-container">$\mathbb{E}[\nabla_\theta \log p(u)] = -1/\theta$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-03T14:30:43.853Z
Answers.Text: does using the gradient estimator not give an answer of $- \frac{1}{2\theta}$? or have I done some maths wrong?
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-03T21:24:11.093Z
Answers.Text: Actually I just redid it and it looks like it should be $-1$. $\nabla_{\theta} \log p(u) = -\frac{1}{\theta}$. $\int_0^{\theta} -\frac{1}{\theta} du = -1$
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T09:34:57.017Z
Answers.Text: No, I think your original answer was correct. I _had_ done some maths wrong. $\mathbb{E}[\nabla_\theta \log p(u)] = \int_0^\theta -\frac{1}{\theta} \times \frac{1}{\theta} du = -\frac{1}{\theta}$
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-03T14:44:46.037Z
Answers.LastActivityDate: 2021-11-03T14:44:46.037Z
Answers.UserDisplayName: Taw
Body: <p>I'm reading the paper <a href="http://proceedings.mlr.press/v32/silver14.pdf" rel="nofollow noreferrer">Deterministic Policy Gradient Algorithms</a>, David Silver et al.</p>
  <p>First of all, in the introduction, the author says that</p>
  <blockquote>
  <p>It was previously believed that the deterministic policy gradient did not exist</p>
  </blockquote>
  <p>But, I wonder why it is. The general version of the policy gradient theorem does not have restrictions about the policy <span class="math-container">$\pi$</span>. So, if we choose the policy <span class="math-container">$\pi$</span> as a Dirac measure, that is, <span class="math-container">$\pi(\cdot | s) = \delta_{a}$</span> for some <span class="math-container">$a \in \mathcal{A}$</span>, then it is exactly the notion of deterministic policy, so we can apply the usual gradient descent theorem.</p>
  <p>Indeed, in theorem 2, they showed that deterministic policy gradient theorem and usual gradient theorem matches when it comes to zero variance. (In fact, I can't understand the statement rigorously, because the policy is something about probability &quot;measure&quot;, and the variance is something about &quot;random variable&quot;.) However, my below computation shows some contradiction.</p>
  <p>Let <span class="math-container">$\pi(\cdot | s) = \delta_a$</span>, a Dirac measure for some atom <span class="math-container">$a \in \mathcal{A}$</span>. Following the notation of the paper DPG, a policy gradient theorem says</p>
  <p><span class="math-container">$$\nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{s, a}[\nabla_{\theta}\log \pi_{\theta}(a | s) * Q^{\pi}(s, a)] $$</span></p>
  <p>A definition of integral shows <span class="math-container">$$\nabla_\theta J(\pi_{\theta}) = \int_{\mathcal{S}, \mathcal{A}} \pi_{\theta}(da|s) \rho^{\pi}{(ds)} \nabla_\theta \log \pi_\theta(a|s) * Q^{\pi}(s,a).
  $$</span>
  But since <span class="math-container">$\pi_\theta$</span> has only an atom at generic element <span class="math-container">$a \in \mathcal{A}$</span>, so it is  same with <span class="math-container">$$\int_\mathcal{S} \rho^{\pi}(s) \nabla_\theta \log \pi_\theta(a | s) * Q^{\pi}(s, a) = \mathbb{E}_s[\nabla_\theta \log \pi_\theta(a|s) * Q^{\pi}(s, a)].$$</span></p>
  <p>(NOTE : the last term seems the same with the first line of the equation, but we get rid of <span class="math-container">$a$</span> from the expectation, by fixing <span class="math-container">$a$</span> corresponding to the atom of <span class="math-container">$s$</span>.) However, note that <span class="math-container">$\log\pi_\theta(a|s) = 1$</span>, since <span class="math-container">$\pi_\theta(a|s) = \delta_a(\{a\}) = 1$</span> as we defined! Thus, <span class="math-container">$\nabla_\theta \log \pi_\theta(a | s) = \nabla_\theta 1 = 0$</span> so we reached that gradient vanishes..</p>
  <p>However, clearly not.</p>
  <p>Can anyone help me?</p>
  
Tags: <reinforcement-learning><deep-rl><papers><policy-gradients><deterministic-pg-theorem>
Title: Why exactly was previously believed that the deterministic policy gradient did not exist?
UserDisplayName: Jiya
ThreadText: # Why exactly was previously believed that the deterministic policy gradient did not exist?
  
  (Asked by: username_0 on 2021-11-03)
  
  I'm reading the paper [Deterministic Policy Gradient Algorithms](http://proceedings.mlr.press/v32/silver14.pdf), David Silver et al.
  
  First of all, in the introduction, the author says that
  
  > It was previously believed that the deterministic policy gradient did not exist
  
  But, I wonder why it is. The general version of the policy gradient theorem does not have restrictions about the policy $\pi$. So, if we choose the policy $\pi$ as a Dirac measure, that is, $\pi(\cdot | s) = \delta\_{a}$ for some $a \in \mathcal{A}$, then it is exactly the notion of deterministic policy, so we can apply the usual gradient descent theorem.
  
  Indeed, in theorem 2, they showed that deterministic policy gradient theorem and usual gradient theorem matches when it comes to zero variance. (In fact, I can't understand the statement rigorously, because the policy is something about probability "measure", and the variance is something about "random variable".) However, my below computation shows some contradiction.
  
  Let $\pi(\cdot | s) = \delta\_a$, a Dirac measure for some atom $a \in \mathcal{A}$. Following the notation of the paper DPG, a policy gradient theorem says
  
  $$\nabla\_{\theta}J(\pi\_{\theta}) = \mathbb{E}\_{s, a}[\nabla\_{\theta}\log \pi\_{\theta}(a | s) \* Q^{\pi}(s, a)] $$
  
  A definition of integral shows $$\nabla\_\theta J(\pi\_{\theta}) = \int\_{\mathcal{S}, \mathcal{A}} \pi\_{\theta}(da|s) \rho^{\pi}{(ds)} \nabla\_\theta \log \pi\_\theta(a|s) \* Q^{\pi}(s,a).
  $$
  But since $\pi\_\theta$ has only an atom at generic element $a \in \mathcal{A}$, so it is same with $$\int\_\mathcal{S} \rho^{\pi}(s) \nabla\_\theta \log \pi\_\theta(a | s) \* Q^{\pi}(s, a) = \mathbb{E}\_s[\nabla\_\theta \log \pi\_\theta(a|s) \* Q^{\pi}(s, a)].$$
  
  (NOTE : the last term seems the same with the first line of the equation, but we get rid of $a$ from the expectation, by fixing $a$ corresponding to the atom of $s$.) However, note that $\log\pi\_\theta(a|s) = 1$, since $\pi\_\theta(a|s) = \delta\_a(\{a\}) = 1$ as we defined! Thus, $\nabla\_\theta \log \pi\_\theta(a | s) = \nabla\_\theta 1 = 0$ so we reached that gradient vanishes..
  
  However, clearly not.
  
  Can anyone help me?
  
  **Tags:** reinforcement-learning, deep-rl, papers, policy-gradients, deterministic-pg-theorem
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-03. Score: 2 | Accepted answer
  
  Actually, your result that the gradient is 0 is correct given your formulation. Indeed, that is why one might have believed that the deterministic policy gradient didn't exist.
  
  The term $\nabla\_{\theta}\log \pi(a \, | \, s)$
  is a type of gradient estimator known as a *likelihood ratio*, and it assumes that the support of $\pi$ does not depend on $\theta$. In other words there must be some non-zero probability of choosing all the possible actions $\mathcal{A}$, and $\theta$ encodes those probabilities. In your idea, $\mathcal{A}$ is a *single action*, and therefore the parameters don't do anything at all. You always have probability 1 of choosing that action. We can't let different parameters choose different actions, as that violates the assumptions of the likelihood ratio and will lead to a biased gradient.
  
  But why can't we let the support of a likelihood ratio depend on $\theta$? I won't prove this in a rigorous way here but you can consider
  \begin{align\*}
  \mathbb{E}\_{y} [ \nabla\_{\theta}\, \log p\_{\theta}(y)] = \mathbb{E}\_{y} \left[ \underset{\epsilon \rightarrow 0}{\lim} \dfrac{\log p\_{\theta+\epsilon}(y) - \log p\_{\theta}(y)}{\epsilon}\right]
  \end{align\*}
  and see that if $p\_{\theta + \epsilon}(y) = 0$, then $\log p\_{\theta+\epsilon}(y) = -\infty$.
  
  I think a more instructive example is to consider what can go wrong if we just go ahead and let the support depend on $\theta$ anyway. Let's consider $\nabla\_{\theta} \, \mathbb{E} [u]$ where $u \sim \text{Uniform}(0, \theta)$ (so $\theta$ is a scalar, and $u$ has support on $[0, \theta ]$). We know $\mathbb{E} [u] = \frac{1}{2}\theta$ so $\nabla\_{\theta}\mathbb{E} [u] = \frac{1}{2}$. You should work out for yourself that the gradient estimator (using a likelihood ratio) gives the utterly wrong result of $\mathbb{E}[\nabla\_\theta \log p(u)] = -1/\theta$
  
  > **Comments:**
  > **username_2**: does using the gradient estimator not give an answer of $- \frac{1}{2\theta}$? or have I done some maths wrong?  
  > **username_1**: Actually I just redid it and it looks like it should be $-1$. $\nabla_{\theta} \log p(u) = -\frac{1}{\theta}$. $\int_0^{\theta} -\frac{1}{\theta} du = -1$  
  > **username_2**: No, I think your original answer was correct. I _had_ done some maths wrong. $\mathbb{E}[\nabla_\theta \log p(u)] = \int_0^\theta -\frac{1}{\theta} \times \frac{1}{\theta} du = -\frac{1}{\theta}$  
  
  ---
  
  