Answers.Body: <p>Well, I'd rather comment, but I don't have yet this privilege, so here are some comments.</p>
  
  <p>First, having a deterministic policy inside the log would do create trivial terms.</p>
  
  <p>Secondly, for me, in Policy Gradient methods, it's a non sense to have a deterministic policy during the optimization, because you want to explore the space of weights. In my experience, you only set the policy to deterministic (in PG method) when you're done with the optimization, and you want to test your network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-06T14:22:54.677Z
Answers.LastActivityDate: 2018-09-06T14:55:21.640Z
Answers.LastEditDate: 2018-09-06T14:55:21.640Z
Answers.UserDisplayName: 16Aghnar
Answers.Body: <p>Here is the gradient that they are discussing in the video:</p>
  <p><span class="math-container">$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right)$$</span></p>
  <p>In this equation, <span class="math-container">$\pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t})$</span> denotes the probability of our policy <span class="math-container">$\pi_{\theta}$</span> selecting the actions <span class="math-container">$\mathbf{a}_{i, t}$</span> that it actually ended up selecting in practice, given the states <span class="math-container">$\mathbf{s}_{i, t}$</span> that it encountered during the episode that we're looking at.</p>
  <p>In the case of a deterministic policy <span class="math-container">$\pi_{\theta}$</span>, we know for sure that the probability of it selecting the actions that it did select must be <span class="math-container">$1$</span> (and the probability of it selecting any other actions would be <span class="math-container">$0$</span>, but such a term does not show up in the equation). So, we have <span class="math-container">$\pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) = 1$</span> for every instance of that term in the above equation. Because <span class="math-container">$\log 1 = 0$</span>, this leads to:</p>
  <p><span class="math-container">\begin{aligned}
  \nabla_{\theta} J(\theta) &amp;\approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \mathbf{s}_{i, t}) \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
  %
  &amp;= \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log 1 \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
  %
  &amp;= \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} 0 \right) \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
  %
  &amp;= \frac{1}{N} \sum_{i=1}^N 0 \left( \sum_{t = 1}^T r(\mathbf{s}_{i,t}, \mathbf{a}_{i, t}) \right) \\
  %
  &amp;= 0 \\
  \end{aligned}</span></p>
  <p>(i.e. you end up with a sum of terms that are all multiplied by <span class="math-container">$0$</span>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-07T01:41:16.330Z
Answers.Text: Just thinking this through, for the case when the policy is stochastic.
  
  If you take a step in the direction of $\nabla_{\theta} \log \pi_{\theta}$, this will increase the probability the policy takes an action -- since the log from (0 -> 1] is decreasingly negative.
  
  Since the gradient is multiplied by the reward, by increasing the probability of an action you increase the expected value of that state-action reward (when the reward is positive, decrease the expected value if it is negative).
  
  Is this correct?
Answers.UserDisplayName: jonperl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-07T02:00:08.787Z
Answers.Text: Ok, yes that is basically what he says at https://youtu.be/XGmd3wcyDg8?t=1295. Thank you again so much -- I couldn't get past that part of the video until I understood the formula.
Answers.UserDisplayName: jonperl
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-07T08:20:27.277Z
Answers.Text: @jonperl Yes that's correct.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-06T14:44:47.477Z
Answers.LastActivityDate: 2022-01-01T13:00:02.367Z
Answers.LastEditDate: 2022-01-01T13:00:02.367Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>In the Berkeley RL class <a href="https://www.youtube.com/watch?v=XGmd3wcyDg8&amp;feature=youtu.be&amp;t=1071" rel="nofollow noreferrer">CS294-112 Fa18 9/5/18</a>, they mention the following gradient would be 0 if the policy is deterministic.</p>
  <p><span class="math-container">$$
  \nabla_{\theta} J(\theta)=E_{\tau \sim \pi_{\theta}(\tau)}\left[\left(\sum_{t=1}^{T} \nabla_{\theta} \log \pi_{\theta}\left(\mathbf{a}_{t} \mid \mathbf{s}_{t}\right)\right)\left(\sum_{t=1}^{T} r\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)\right)\right]
  $$</span></p>
  <p>Why is that?</p>
  
Tags: <reinforcement-learning><policy-gradients><policies><gradient><calculus>
Title: Why is the derivative of this objective function 0 if the policy is deterministic?
UserDisplayName: jonperl
ThreadText: # Why is the derivative of this objective function 0 if the policy is deterministic?
  
  (Asked by: username_0 on 2018-09-06)
  
  In the Berkeley RL class [CS294-112 Fa18 9/5/18](https://www.youtube.com/watch?v=XGmd3wcyDg8&feature=youtu.be&t=1071), they mention the following gradient would be 0 if the policy is deterministic.
  
  $$
  \nabla\_{\theta} J(\theta)=E\_{\tau \sim \pi\_{\theta}(\tau)}\left[\left(\sum\_{t=1}^{T} \nabla\_{\theta} \log \pi\_{\theta}\left(\mathbf{a}\_{t} \mid \mathbf{s}\_{t}\right)\right)\left(\sum\_{t=1}^{T} r\left(\mathbf{s}\_{t}, \mathbf{a}\_{t}\right)\right)\right]
  $$
  
  Why is that?
  
  **Tags:** reinforcement-learning, policy-gradients, policies, gradient, calculus
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-09-06. Score: 13 | Accepted answer
  
  Here is the gradient that they are discussing in the video:
  
  $$\nabla\_{\theta} J(\theta) \approx \frac{1}{N} \sum\_{i=1}^N \left( \sum\_{t=1}^T \nabla\_{\theta} \log \pi\_{\theta} (\mathbf{a}\_{i, t} \vert \mathbf{s}\_{i, t}) \right) \left( \sum\_{t = 1}^T r(\mathbf{s}\_{i,t}, \mathbf{a}\_{i, t}) \right)$$
  
  In this equation, $\pi\_{\theta} (\mathbf{a}\_{i, t} \vert \mathbf{s}\_{i, t})$ denotes the probability of our policy $\pi\_{\theta}$ selecting the actions $\mathbf{a}\_{i, t}$ that it actually ended up selecting in practice, given the states $\mathbf{s}\_{i, t}$ that it encountered during the episode that we're looking at.
  
  In the case of a deterministic policy $\pi\_{\theta}$, we know for sure that the probability of it selecting the actions that it did select must be $1$ (and the probability of it selecting any other actions would be $0$, but such a term does not show up in the equation). So, we have $\pi\_{\theta} (\mathbf{a}\_{i, t} \vert \mathbf{s}\_{i, t}) = 1$ for every instance of that term in the above equation. Because $\log 1 = 0$, this leads to:
  
  \begin{aligned}
  \nabla\_{\theta} J(\theta) &\approx \frac{1}{N} \sum\_{i=1}^N \left( \sum\_{t=1}^T \nabla\_{\theta} \log \pi\_{\theta} (\mathbf{a}\_{i, t} \vert \mathbf{s}\_{i, t}) \right) \left( \sum\_{t = 1}^T r(\mathbf{s}\_{i,t}, \mathbf{a}\_{i, t}) \right) \\
  %
  &= \frac{1}{N} \sum\_{i=1}^N \left( \sum\_{t=1}^T \nabla\_{\theta} \log 1 \right) \left( \sum\_{t = 1}^T r(\mathbf{s}\_{i,t}, \mathbf{a}\_{i, t}) \right) \\
  %
  &= \frac{1}{N} \sum\_{i=1}^N \left( \sum\_{t=1}^T \nabla\_{\theta} 0 \right) \left( \sum\_{t = 1}^T r(\mathbf{s}\_{i,t}, \mathbf{a}\_{i, t}) \right) \\
  %
  &= \frac{1}{N} \sum\_{i=1}^N 0 \left( \sum\_{t = 1}^T r(\mathbf{s}\_{i,t}, \mathbf{a}\_{i, t}) \right) \\
  %
  &= 0 \\
  \end{aligned}
  
  (i.e. you end up with a sum of terms that are all multiplied by $0$).
  
  > **Comments:**
  > **username_0**: Just thinking this through, for the case when the policy is stochastic.  If you take a step in the direction of $\nabla_{\theta} \log \pi_{\theta}$, this will increase the probability the policy takes an action -- since the log from (0 -> 1] is decreasingly negative.  Since the gradient is multiplied by the reward, by increasing the probability of an action you increase the expected value of that state-action reward (when the reward is positive, decrease the expected value if it is negative).  Is this correct?  
  > **username_0**: Ok, yes that is basically what he says at https://youtu.be/XGmd3wcyDg8?t=1295. Thank you again so much -- I couldn't get past that part of the video until I understood the formula.  
  > **username_2**: @username_0 Yes that's correct.  
  
  ---
  
  ### Answer by username_1 on 2018-09-06. Score: 3
  
  Well, I'd rather comment, but I don't have yet this privilege, so here are some comments.
  
  First, having a deterministic policy inside the log would do create trivial terms.
  
  Secondly, for me, in Policy Gradient methods, it's a non sense to have a deterministic policy during the optimization, because you want to explore the space of weights. In my experience, you only set the policy to deterministic (in PG method) when you're done with the optimization, and you want to test your network.
  
  
  ---
  
  