Answers.Body: <p>In the policy gradient theorem, we don't need to write <span class="math-container">$r$</span> as a function of <span class="math-container">$a$</span> because the only time we explicitly 'see' <span class="math-container">$r$</span> is when we are taking the expectation with respect to the policy. For the first couple lines of the PG theorem we have
  <span class="math-container">\begin{align}
  \nabla v_\pi(s) &amp;= \nabla \left[ \sum_a \pi(a|s) q_\pi (s,a) \right] \;, \\ &amp;= \sum_a \left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s) \nabla\sum_{s',r} p(s',r|s,a)(r+ v_\pi(s')) \right] \; ;
  \end{align}</span>
  you can see that we are taking expectation of <span class="math-container">$r$</span> with respect to the policy, so we don't need to write something like <span class="math-container">$r(s,\pi(a|s))$</span> (especially because this notation doesn't really make sense for a stochastic policy). This is why we don't need to take the derivative of <span class="math-container">$r$</span> with respect to the policy parameters. Now, the next line of the PG theorem is
  <span class="math-container">$$\nabla v_\pi(s) = \sum_a \left[ \nabla \pi(a|s) q_\pi(s,a) + \pi(a|s)\sum_{s'} p(s'|s,a) \nabla v_\pi(s') \right] \; ;$$</span>
  so now we have an equation similar to the bellman equation in terms of the <span class="math-container">$\nabla v_\pi(s)$</span>'s, so we can unroll this repeatedly meaning we never have to take an explicit derivative of the value function.</p>
  <p>For the deterministic gradient, this is a bit different. In general we have
  <span class="math-container">$$v_\pi(s) = \mathbb{E}_\pi[Q(s,a)] = \sum_a \pi(a|s) Q(s,a)\;,$$</span>
  so for a deterministic policy (denoted by <span class="math-container">$\pi(s)$</span> which represents the action taken in state <span class="math-container">$s$</span>) this becomes
  <span class="math-container">$$v_\pi(s) = Q(s,\pi(s))$$</span>
  because the deterministic policy has 0 probability for all actions except one, where it has probability one.</p>
  <p>Now, in the deterministic policy gradient theorem we can write
  <span class="math-container">$$\nabla v_\pi(s) = \nabla Q(s,\pi(s)) = \nabla \left(r(s, \pi(s)) + \sum_{s'} p(s'|s,a)v(s') \right)\;.$$</span></p>
  <p>We have to write <span class="math-container">$r$</span> explicitly as a function of <span class="math-container">$s,a$</span> now because we are not taking an expectation with respect to the actions because we have a deterministic policy. Now, if you replace where I have written <span class="math-container">$\nabla$</span> with the notation you have used for the derivatives you will arrive at the same result and you'll see why you need to use the chain rule, which I believe you understand because your question was more why <em>don't</em> we use the chain rule for the normal policy gradient, which I have hopefully explained -- it is essentially because of how an expectation over the action space works with a deterministic policy vs. a stochastic policy.</p>
  <p>Another way to think of this is as follows -- the term you're concerned with is obtained by expanding <span class="math-container">$\nabla q_\pi(s,a) = \nabla \sum_{s', r}p(s',r|s,a)(r(s,a) + v_\pi(s'))$</span>. Because, by definition of the <span class="math-container">$Q$</span> function, we have conditioned on knowing <span class="math-container">$a,s$</span> then <span class="math-container">$a$</span> is completely independent of the policy in this scenario - we could even condition on an action that the policy would have 0 probability for - thus the derivative of <span class="math-container">$r$</span> with respect to the policy parameters is 0.</p>
  <p>However, in the deterministic policy gradient we are taking <span class="math-container">$\nabla q_\pi(s, \pi(s)) = \nabla \left(r(s, \pi(s)) + \sum_{s'} p(s'|s,a) v_\pi(s')\right)$</span> -- here <span class="math-container">$r$</span> clearly depends on the policy parameters because the action taken was the deterministic action given by the policy in the state <span class="math-container">$s$</span>, thus the derivative wrt the policy parameters is not necessarily 0!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-05T16:41:14.963Z
Answers.Text: Thank you very much David! Your answer is extremely helpful to me. I mark the answer as accepted. Nevertheless, I still do not fully comprehend your answer - most likely because I am lacking the knowledge about how conditional expectations interact with derivatives. I understand your argument as saying that one cannot compute $\partial E[y(x) | x] / \partial x$? If, by any chance, you know a good book for how conditional expectations affect derivatives, I'd be happy to read that! Anyway - thanks again for the detailed answer!
Answers.UserDisplayName: fabian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-05T18:06:10.567Z
Answers.Text: it is not so much that we can't compute that derivate, in fact in general this derivative is possible to calculate. It is more that in the stochastic policy gradient the policy affects the reward through the expectation, if this makes sense. However, in the deterministic policy case, you do technically still take the same expectation it is just because of how deterministic expectations work out that it directly effects the reward, which is what I tried to stress in my answer.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-05T18:13:26.670Z
Answers.Text: @fabian also a good way to think of it is that in the standard policy gradient $r$ is a function of $s,a$ (denote this by $r(s,a)$) but we condition on knowing $a$ so this is independent of the policy, thus it is constant wrt policy parameters. In the deterministic PG we still condition on taking an action $a$ but here the action is $a=\pi(s)$ which shows there is a clear dependence on the policy parameters. Please let me know if this helps clarify things and I can edit it into my answer, as I feel my original explanation may have been a bit convoluted.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T10:54:28.373Z
Answers.Text: Thanks for your comments! I benefited particularly from your first comment. In the second comment, we could think of the stochastic policy just as the deterministic policy with a little bit of noise (as the behavioral exploration policy in DDPG) so there is also a *clear dependence on the policy parameters* as you said. Overall, its still a bit difficult for me to understand the precise mathematical reason / proof for the differences in the two theorems but your answers and comments illuminated many aspects greatly for me; thank you again for that.
Answers.UserDisplayName: fabian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T13:21:51.363Z
Answers.Text: @fabian no, not really. the deterministic policy is a limiting case for a stochastic policy, as shown in the deterministic policy gradient paper. If you read my edit on the paper, then it should clarify things further. Because we have conditioned on some $a$ in the stochastic case, there is no dependency on the policy, because as I mentioned it is just an arbitrary action - when this is done in the deterministic case there is absolutely a dependency because we are conditioning _on the action the deterministic policy gives_.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T15:08:32.497Z
Answers.Text: Your last comment (and also the last two paragraphs of your answer) brought the point home. This really makes sense. Thank you very much indeed!
Answers.UserDisplayName: fabian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T16:04:08.893Z
Answers.Text: Just to summarize and check whether I understood it: In the stochastic case (and under conditional expectation in general) $E[r|s,a]=f(s,a)$ for some function $f$. Because $a$ is stochastic $a \sim \pi(s; \theta)$ and because we have already conditioned on $a$ we *cannot go back*  and take the derivative w.r.t. $\theta$. In contrast for the deterministic case $E[r|s]=f(s) = f(\pi(s; \theta))$ we can take the derivative w.r.t $\theta$ as the entire chain is deterministic. Could I summarize your argument like this, @David?
Answers.UserDisplayName: fabian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-06T16:20:10.027Z
Answers.Text: Yes, exactly. Good job!
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-19T17:23:39.527Z
Answers.Text: @David thank you for your answer, but could you elaborate on notation I am still unfamiliar with ? I can't find corresponding parts in the proof or justification for some of your statements. Eg.: (a) $(r + v_{\pi}(s'))$ akin more to the Bellman-equ. (2nd equ-line) never appears in the proof and I can't find it by basic rewriting of the equations (yet considering $\mathcal{P}_{ss'}^a$). (b) The deterministic optimal state value function $v_{\pi}(s)$ is the expected value (under policy $\pi$?) of all action value functions but just a sum of values? (c) Why condition on the action not state $s$?
Answers.UserDisplayName: BadAtLaTeX
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-05T10:23:24.770Z
Answers.LastActivityDate: 2020-08-06T10:01:15.673Z
Answers.LastEditDate: 2020-08-06T10:01:15.673Z
Answers.UserDisplayName: David
Body: <p>I would like to understand the difference between the <a href="https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf" rel="nofollow noreferrer">standard policy gradient theorem</a> and the <a href="http://proceedings.mlr.press/v32/silver14-supp.pdf" rel="nofollow noreferrer">deterministic policy gradient theorem</a>. These two theorem are quite different, although the <em>only</em> difference is whether the policy function is deterministic or stochastic.  I summarized the relevant steps of the theorems below. The policy function is <span class="math-container">$\pi$</span> which has parameters <span class="math-container">$\theta$</span>.</p>
  <p><strong>Standard Policy Gradient</strong>
  <span class="math-container">$$
  \begin{aligned}
  \dfrac{\partial V}{\partial \theta} &amp;= \dfrac{\partial}{\partial \theta} \left[ \sum_a \pi(a|s) Q(a,s) \right]  \\
  &amp;= \sum_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta}  Q(a,s) + \pi(a|s) \dfrac{\partial Q(a,s)}{\partial \theta} \right] \\
  &amp;= \sum_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta}  Q(a,s) + \pi(a|s) \dfrac{\partial}{\partial \theta} \left[ R + \sum_{s'} \gamma p(s'|s,a) V(s') \right] \right]  \\
  &amp;= \sum_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta}  Q(a,s) + \pi(a|s) \gamma \sum_{s'} p(s'|s,a) \dfrac{\partial V(s') }{\partial \theta} \right]
  \end{aligned}
  $$</span>
  When one now expands next period's value function <span class="math-container">$V(s')$</span> again one can eventually reach the final policy gradient:
  <span class="math-container">$$
  \dfrac{\partial J}{\partial \theta} = \sum_s \rho(s) \sum_a \dfrac{\pi(a|s)}{\partial \theta}  Q(s,a)
  $$</span>
  with <span class="math-container">$\rho$</span> being the stationary distribution. What I find particularly interesting is that there is no derivative of <span class="math-container">$R$</span> with respect to <span class="math-container">$\theta$</span> and also not of the probability distribution <span class="math-container">$p(s'|s,a)$</span> with respect to <span class="math-container">$\theta$</span>. The derivation of the deterministic policy gradient theorem is different:</p>
  <p><strong>Deterministic Policy Gradient Theorem</strong>
  <span class="math-container">$$
  \begin{aligned}
  \dfrac{\partial V}{\partial \theta} &amp;= \dfrac{\partial}{\partial \theta} Q(\pi(s),s)  \\
  &amp;= \dfrac{\partial}{\partial \theta} \left[ R(s, \pi(s)) + \gamma \sum_{s'} p(s'|a,s) V(s') \right] \\
  &amp;= \dfrac{R(s, a)}{\partial a}\dfrac{\pi(s)}{\partial \theta} + \dfrac{\partial}{\partial \theta} \left[\gamma \sum_{s'} p(s'|a,s) V(s')   \right]  \\
  &amp;= \dfrac{R(s, a)}{\partial a}\dfrac{\pi(s)}{\partial \theta} + \gamma \sum_{s'}  \left[p(s'|\mu(s),s) \dfrac{V(s')}{\partial \theta} + \dfrac{\pi(s)}{\partial \theta} \dfrac{p(s'|s,a)}{\partial a} V(s')   \right]  \\
  &amp;= \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial}{\partial a} \left[ R(s, a) +  p(s'|s,a) V(s') \right] +  \gamma  p(s'|\pi(s),s) \dfrac{V(s')}{\partial \theta}  \\
  &amp;= \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial Q(s, a)}{\partial a} +  \gamma  p(s'|\pi(s),s) \dfrac{V(s')}{\partial \theta}  \\
  \end{aligned}
  $$</span>
  Again, one can obtain the finaly policy gradient by expanding next period's value function. The policy gradient is:
  <span class="math-container">$$
  \dfrac{\partial J}{\partial \theta} = \sum_s \rho(s) \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial Q(s,a))}{\partial a}
  $$</span>
  In contrast to the standard policy gradient, the equations contain derivatives of the reward function <span class="math-container">$R$</span> and the conditional probability <span class="math-container">$p(s'|s, a,)$</span> with respect to <span class="math-container">$a$</span>.</p>
  <p><strong>Question</strong></p>
  <p>Why do the two theorems differ in their treatment of the derivatives of <span class="math-container">$R$</span> and the conditional probability? Does determinism in the policy function make such a difference for the derivatives?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-04T18:17:34.533Z
Comments.Text: I don't have time to leave a full answer for now but in short it is because the latter requires the use of a chain rule because the parameters we differentiate with respect to are parameters of $\pi$ but we have access only to $Q$ -- it is like wanting to do the following $\frac{d}{dx} f(g(x))$. I can try and leave a better answer if nobody else has when I get chance.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-05T05:53:25.980Z
Comments.Text: Thank you @DavidIreland for your comment! I'd be very grateful to read your full answer! Can you elaborate a bit why the chain rule should not be applicable in the usual policy gradient theorem? Implicitly, $R(s,a)$ is also function of $\pi(a|s)$, although a stochastic function.
Comments.UserDisplayName: fabian
Tags: <reinforcement-learning><policy-gradients><policy-gradient-theorem><deterministic-pg-theorem><theorems>
Title: Why do the standard and deterministic Policy Gradient Theorems differ in their treatment of the derivatives of $R$ and the conditional probability?
UserDisplayName: fabian
ThreadText: # Why do the standard and deterministic Policy Gradient Theorems differ in their treatment of the derivatives of $R$ and the conditional probability?
  
  (Asked by: username_0 on 2020-08-04)
  
  I would like to understand the difference between the [standard policy gradient theorem](https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf) and the [deterministic policy gradient theorem](http://proceedings.mlr.press/v32/silver14-supp.pdf). These two theorem are quite different, although the *only* difference is whether the policy function is deterministic or stochastic. I summarized the relevant steps of the theorems below. The policy function is $\pi$ which has parameters $\theta$.
  
  **Standard Policy Gradient**
  $$
  \begin{aligned}
  \dfrac{\partial V}{\partial \theta} &= \dfrac{\partial}{\partial \theta} \left[ \sum\_a \pi(a|s) Q(a,s) \right] \\
  &= \sum\_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta} Q(a,s) + \pi(a|s) \dfrac{\partial Q(a,s)}{\partial \theta} \right] \\
  &= \sum\_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta} Q(a,s) + \pi(a|s) \dfrac{\partial}{\partial \theta} \left[ R + \sum\_{s'} \gamma p(s'|s,a) V(s') \right] \right] \\
  &= \sum\_a \left[ \dfrac{\partial \pi(a|s)}{\partial \theta} Q(a,s) + \pi(a|s) \gamma \sum\_{s'} p(s'|s,a) \dfrac{\partial V(s') }{\partial \theta} \right]
  \end{aligned}
  $$
  When one now expands next period's value function $V(s')$ again one can eventually reach the final policy gradient:
  $$
  \dfrac{\partial J}{\partial \theta} = \sum\_s \rho(s) \sum\_a \dfrac{\pi(a|s)}{\partial \theta} Q(s,a)
  $$
  with $\rho$ being the stationary distribution. What I find particularly interesting is that there is no derivative of $R$ with respect to $\theta$ and also not of the probability distribution $p(s'|s,a)$ with respect to $\theta$. The derivation of the deterministic policy gradient theorem is different:
  
  **Deterministic Policy Gradient Theorem**
  $$
  \begin{aligned}
  \dfrac{\partial V}{\partial \theta} &= \dfrac{\partial}{\partial \theta} Q(\pi(s),s) \\
  &= \dfrac{\partial}{\partial \theta} \left[ R(s, \pi(s)) + \gamma \sum\_{s'} p(s'|a,s) V(s') \right] \\
  &= \dfrac{R(s, a)}{\partial a}\dfrac{\pi(s)}{\partial \theta} + \dfrac{\partial}{\partial \theta} \left[\gamma \sum\_{s'} p(s'|a,s) V(s') \right] \\
  &= \dfrac{R(s, a)}{\partial a}\dfrac{\pi(s)}{\partial \theta} + \gamma \sum\_{s'} \left[p(s'|\mu(s),s) \dfrac{V(s')}{\partial \theta} + \dfrac{\pi(s)}{\partial \theta} \dfrac{p(s'|s,a)}{\partial a} V(s') \right] \\
  &= \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial}{\partial a} \left[ R(s, a) + p(s'|s,a) V(s') \right] + \gamma p(s'|\pi(s),s) \dfrac{V(s')}{\partial \theta} \\
  &= \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial Q(s, a)}{\partial a} + \gamma p(s'|\pi(s),s) \dfrac{V(s')}{\partial \theta} \\
  \end{aligned}
  $$
  Again, one can obtain the finaly policy gradient by expanding next period's value function. The policy gradient is:
  $$
  \dfrac{\partial J}{\partial \theta} = \sum\_s \rho(s) \dfrac{\pi(s)}{\partial \theta} \dfrac{\partial Q(s,a))}{\partial a}
  $$
  In contrast to the standard policy gradient, the equations contain derivatives of the reward function $R$ and the conditional probability $p(s'|s, a,)$ with respect to $a$.
  
  **Question**
  
  Why do the two theorems differ in their treatment of the derivatives of $R$ and the conditional probability? Does determinism in the policy function make such a difference for the derivatives?
  
  **Tags:** reinforcement-learning, policy-gradients, policy-gradient-theorem, deterministic-pg-theorem, theorems
  
  > **Comments:**
  > **username_1**: I don't have time to leave a full answer for now but in short it is because the latter requires the use of a chain rule because the parameters we differentiate with respect to are parameters of $\pi$ but we have access only to $Q$ -- it is like wanting to do the following $\frac{d}{dx} f(g(x))$. I can try and leave a better answer if nobody else has when I get chance.  
  > **username_0**: Thank you @DavidIreland for your comment! I'd be very grateful to read your full answer! Can you elaborate a bit why the chain rule should not be applicable in the usual policy gradient theorem? Implicitly, $R(s,a)$ is also function of $\pi(a|s)$, although a stochastic function.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-05. Score: 6 | Accepted answer
  
  In the policy gradient theorem, we don't need to write $r$ as a function of $a$ because the only time we explicitly 'see' $r$ is when we are taking the expectation with respect to the policy. For the first couple lines of the PG theorem we have
  \begin{align}
  \nabla v\_\pi(s) &= \nabla \left[ \sum\_a \pi(a|s) q\_\pi (s,a) \right] \;, \\ &= \sum\_a \left[ \nabla \pi(a|s) q\_\pi(s,a) + \pi(a|s) \nabla\sum\_{s',r} p(s',r|s,a)(r+ v\_\pi(s')) \right] \; ;
  \end{align}
  you can see that we are taking expectation of $r$ with respect to the policy, so we don't need to write something like $r(s,\pi(a|s))$ (especially because this notation doesn't really make sense for a stochastic policy). This is why we don't need to take the derivative of $r$ with respect to the policy parameters. Now, the next line of the PG theorem is
  $$\nabla v\_\pi(s) = \sum\_a \left[ \nabla \pi(a|s) q\_\pi(s,a) + \pi(a|s)\sum\_{s'} p(s'|s,a) \nabla v\_\pi(s') \right] \; ;$$
  so now we have an equation similar to the bellman equation in terms of the $\nabla v\_\pi(s)$'s, so we can unroll this repeatedly meaning we never have to take an explicit derivative of the value function.
  
  For the deterministic gradient, this is a bit different. In general we have
  $$v\_\pi(s) = \mathbb{E}\_\pi[Q(s,a)] = \sum\_a \pi(a|s) Q(s,a)\;,$$
  so for a deterministic policy (denoted by $\pi(s)$ which represents the action taken in state $s$) this becomes
  $$v\_\pi(s) = Q(s,\pi(s))$$
  because the deterministic policy has 0 probability for all actions except one, where it has probability one.
  
  Now, in the deterministic policy gradient theorem we can write
  $$\nabla v\_\pi(s) = \nabla Q(s,\pi(s)) = \nabla \left(r(s, \pi(s)) + \sum\_{s'} p(s'|s,a)v(s') \right)\;.$$
  
  We have to write $r$ explicitly as a function of $s,a$ now because we are not taking an expectation with respect to the actions because we have a deterministic policy. Now, if you replace where I have written $\nabla$ with the notation you have used for the derivatives you will arrive at the same result and you'll see why you need to use the chain rule, which I believe you understand because your question was more why *don't* we use the chain rule for the normal policy gradient, which I have hopefully explained -- it is essentially because of how an expectation over the action space works with a deterministic policy vs. a stochastic policy.
  
  Another way to think of this is as follows -- the term you're concerned with is obtained by expanding $\nabla q\_\pi(s,a) = \nabla \sum\_{s', r}p(s',r|s,a)(r(s,a) + v\_\pi(s'))$. Because, by definition of the $Q$ function, we have conditioned on knowing $a,s$ then $a$ is completely independent of the policy in this scenario - we could even condition on an action that the policy would have 0 probability for - thus the derivative of $r$ with respect to the policy parameters is 0.
  
  However, in the deterministic policy gradient we are taking $\nabla q\_\pi(s, \pi(s)) = \nabla \left(r(s, \pi(s)) + \sum\_{s'} p(s'|s,a) v\_\pi(s')\right)$ -- here $r$ clearly depends on the policy parameters because the action taken was the deterministic action given by the policy in the state $s$, thus the derivative wrt the policy parameters is not necessarily 0!
  
  > **Comments:**
  > **username_0**: Thank you very much username_1! Your answer is extremely helpful to me. I mark the answer as accepted. Nevertheless, I still do not fully comprehend your answer - most likely because I am lacking the knowledge about how conditional expectations interact with derivatives. I understand your argument as saying that one cannot compute $\partial E[y(x) | x] / \partial x$? If, by any chance, you know a good book for how conditional expectations affect derivatives, I'd be happy to read that! Anyway - thanks again for the detailed answer!  
  > **username_1**: it is not so much that we can't compute that derivate, in fact in general this derivative is possible to calculate. It is more that in the stochastic policy gradient the policy affects the reward through the expectation, if this makes sense. However, in the deterministic policy case, you do technically still take the same expectation it is just because of how deterministic expectations work out that it directly effects the reward, which is what I tried to stress in my answer.  
  > **username_1**: @username_0 also a good way to think of it is that in the standard policy gradient $r$ is a function of $s,a$ (denote this by $r(s,a)$) but we condition on knowing $a$ so this is independent of the policy, thus it is constant wrt policy parameters. In the deterministic PG we still condition on taking an action $a$ but here the action is $a=\pi(s)$ which shows there is a clear dependence on the policy parameters. Please let me know if this helps clarify things and I can edit it into my answer, as I feel my original explanation may have been a bit convoluted.  
  > **username_0**: Thanks for your comments! I benefited particularly from your first comment. In the second comment, we could think of the stochastic policy just as the deterministic policy with a little bit of noise (as the behavioral exploration policy in DDPG) so there is also a *clear dependence on the policy parameters* as you said. Overall, its still a bit difficult for me to understand the precise mathematical reason / proof for the differences in the two theorems but your answers and comments illuminated many aspects greatly for me; thank you again for that.  
  > **username_1**: @username_0 no, not really. the deterministic policy is a limiting case for a stochastic policy, as shown in the deterministic policy gradient paper. If you read my edit on the paper, then it should clarify things further. Because we have conditioned on some $a$ in the stochastic case, there is no dependency on the policy, because as I mentioned it is just an arbitrary action - when this is done in the deterministic case there is absolutely a dependency because we are conditioning _on the action the deterministic policy gives_.  
  > **username_0**: Your last comment (and also the last two paragraphs of your answer) brought the point home. This really makes sense. Thank you very much indeed!  
  > **username_0**: Just to summarize and check whether I understood it: In the stochastic case (and under conditional expectation in general) $E[r|s,a]=f(s,a)$ for some function $f$. Because $a$ is stochastic $a \sim \pi(s; \theta)$ and because we have already conditioned on $a$ we *cannot go back*  and take the derivative w.r.t. $\theta$. In contrast for the deterministic case $E[r|s]=f(s) = f(\pi(s; \theta))$ we can take the derivative w.r.t $\theta$ as the entire chain is deterministic. Could I summarize your argument like this, @username_1?  
  > **username_1**: Yes, exactly. Good job!  
  > **username_2**: @username_1 thank you for your answer, but could you elaborate on notation I am still unfamiliar with ? I can't find corresponding parts in the proof or justification for some of your statements. Eg.: (a) $(r + v_{\pi}(s'))$ akin more to the Bellman-equ. (2nd equ-line) never appears in the proof and I can't find it by basic rewriting of the equations (yet considering $\mathcal{P}_{ss'}^a$). (b) The deterministic optimal state value function $v_{\pi}(s)$ is the expected value (under policy $\pi$?) of all action value functions but just a sum of values? (c) Why condition on the action not state $s$?  
  
  ---
  
  