Answers.Body: <p>There is no sign error and we should not change to <span class="math-container">$\arg\max$</span>. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.</p>
  <p>In short, we want to first find the derivative of the RL objective <span class="math-container">$J(\theta) = v_\pi(s_0)$</span>, where <span class="math-container">$\pi$</span> is our policy that depends on some parameters <span class="math-container">$\theta$</span>. The policy gradient theorem tells us that
  <span class="math-container">$$\nabla_\theta J(\theta) = \mathbb{E}_{a\sim\pi, s\sim\mu}\left[ G_t \nabla_\theta \log\pi(A_t|S_t)\right]\;;$$</span>
  where <span class="math-container">$\mu$</span> is our state distribution induced by the policy <span class="math-container">$\pi$</span> and <span class="math-container">$G_t$</span> are the discounted returns.</p>
  <p>Now, as we want to <em>maximise</em> our objective <span class="math-container">$J(\theta)$</span>, we want to perform gradient <em>ascent</em>. That is, our parameters should be updated according to
  <span class="math-container">$$\theta_{t+1} = \theta_t + \alpha \nabla_\theta J(\theta)\;.$$</span></p>
  <p>The reason that you will often see the update written as a minimisation problem is because most software is built to minimise functions, rather than maximise. This is not a problem though as finding the maximum of <span class="math-container">$f$</span> is just the same as finding the minimum of <span class="math-container">$-f$</span>. Using this, we can then say our parameters can be updated using gradient <em>descent</em> on the negative of our objective, i.e. we get our parameter update rule to be
  <span class="math-container">$$\theta_{t+1} = \theta_t - \alpha \nabla_\theta (-J(\theta))\;;$$</span>
  which is exactly what you have in the book.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-08T09:23:25.027Z
Answers.LastActivityDate: 2021-04-08T09:23:25.027Z
Answers.UserDisplayName: David
Body: <p>I don't understand the policy gradient as explained in <strong>Chapter-9</strong> (<em>Deep Reinforcement Learning</em>) of the book <strong>Fundamentals of deep learning</strong>.</p>
  <p>Here is the whole paragraph:</p>
  <h2><strong>Policy Learning via Policy Gradients</strong></h2>
  <blockquote>
  <p>In typical supervised learning, we can use stochastic gradient descent to update our parameters to minimize the loss computed from our network's output and the true label. We are optimizing the expression:
  <span class="math-container">$$
  \arg \min _{\theta} \Sigma_{i} \log p\left(y_{i} \mid x_{i} ; \theta\right)
  $$</span>
  In reinforcement learning, we don't have a true label, only reward signals. However, we can still use SGD to optimize our weights using something called policy gradients. We can use the actions the agent takes, and the returns associated with those actions, to encourage our model weights to take good actions that lead to high reward, and to avoid bad ones that lead to low reward. The expression we optimize for is:
  <span class="math-container">$$
  \arg \min _{\theta}-\sum_{i} R_{i} \log p\left(y_{i} \mid x_{i} ; \theta\right)
  $$</span>
  where <span class="math-container">$y_{i}$</span> is the action taken by the agent at time step <span class="math-container">$t$</span> and where <span class="math-container">$R_{i}$</span> is our discounted future return. A In this way, we scale our loss by the value of our return, so if the model chose an action that led to negative return, this would lead to greater loss. Furthermore, if the model is very confident in that bad decision, it would get penalized even more, since we are taking into account the log probability of the model choosing that action. With our loss function defined, we can apply SGD to minimize our loss and learn a good policy.</p>
  </blockquote>
  <p>The first expression about the loss computed in a network already seems false since the log of a probability is always negative, and taking the <span class="math-container">$θ$</span> (weights) for which the expression is minimal doesn't seem right because it would favor very unsure answers.</p>
  <p>The same goes with the next expression on policy gradient. A very negative <span class="math-container">$R_i$</span> and very unsure <span class="math-container">$p(y_i)$</span> would both be big negatives and multiplied together give a big positive value. Since there is a - sign in front of the expression, this would be the best configuration for the <code>argmin</code>. Meaning we are looking for weights in the policy that give highly negative rewards and for highly unsure actions. This just doesn't make sense to me.</p>
  <p>Is it just a sign error (or we could just change to <code>argmax</code>)? Or is there more to it?</p>
  
Tags: <deep-rl><objective-functions><math><policy-gradients><policies>
Title: Is the policy gradient expression in Fundamentals of Deep Learning wrong?
UserDisplayName: Jonathan Sands
ThreadText: # Is the policy gradient expression in Fundamentals of Deep Learning wrong?
  
  (Asked by: username_0 on 2021-04-08)
  
  I don't understand the policy gradient as explained in **Chapter-9** (*Deep Reinforcement Learning*) of the book **Fundamentals of deep learning**.
  
  Here is the whole paragraph:
  
  ## **Policy Learning via Policy Gradients**
  
  > In typical supervised learning, we can use stochastic gradient descent to update our parameters to minimize the loss computed from our network's output and the true label. We are optimizing the expression:
  > $$
  > \arg \min \_{\theta} \Sigma\_{i} \log p\left(y\_{i} \mid x\_{i} ; \theta\right)
  > $$
  > In reinforcement learning, we don't have a true label, only reward signals. However, we can still use SGD to optimize our weights using something called policy gradients. We can use the actions the agent takes, and the returns associated with those actions, to encourage our model weights to take good actions that lead to high reward, and to avoid bad ones that lead to low reward. The expression we optimize for is:
  > $$
  > \arg \min \_{\theta}-\sum\_{i} R\_{i} \log p\left(y\_{i} \mid x\_{i} ; \theta\right)
  > $$
  > where $y\_{i}$ is the action taken by the agent at time step $t$ and where $R\_{i}$ is our discounted future return. A In this way, we scale our loss by the value of our return, so if the model chose an action that led to negative return, this would lead to greater loss. Furthermore, if the model is very confident in that bad decision, it would get penalized even more, since we are taking into account the log probability of the model choosing that action. With our loss function defined, we can apply SGD to minimize our loss and learn a good policy.
  
  The first expression about the loss computed in a network already seems false since the log of a probability is always negative, and taking the $θ$ (weights) for which the expression is minimal doesn't seem right because it would favor very unsure answers.
  
  The same goes with the next expression on policy gradient. A very negative $R\_i$ and very unsure $p(y\_i)$ would both be big negatives and multiplied together give a big positive value. Since there is a - sign in front of the expression, this would be the best configuration for the `argmin`. Meaning we are looking for weights in the policy that give highly negative rewards and for highly unsure actions. This just doesn't make sense to me.
  
  Is it just a sign error (or we could just change to `argmax`)? Or is there more to it?
  
  **Tags:** deep-rl, objective-functions, math, policy-gradients, policies
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-08. Score: 1
  
  There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.
  
  In short, we want to first find the derivative of the RL objective $J(\theta) = v\_\pi(s\_0)$, where $\pi$ is our policy that depends on some parameters $\theta$. The policy gradient theorem tells us that
  $$\nabla\_\theta J(\theta) = \mathbb{E}\_{a\sim\pi, s\sim\mu}\left[ G\_t \nabla\_\theta \log\pi(A\_t|S\_t)\right]\;;$$
  where $\mu$ is our state distribution induced by the policy $\pi$ and $G\_t$ are the discounted returns.
  
  Now, as we want to *maximise* our objective $J(\theta)$, we want to perform gradient *ascent*. That is, our parameters should be updated according to
  $$\theta\_{t+1} = \theta\_t + \alpha \nabla\_\theta J(\theta)\;.$$
  
  The reason that you will often see the update written as a minimisation problem is because most software is built to minimise functions, rather than maximise. This is not a problem though as finding the maximum of $f$ is just the same as finding the minimum of $-f$. Using this, we can then say our parameters can be updated using gradient *descent* on the negative of our objective, i.e. we get our parameter update rule to be
  $$\theta\_{t+1} = \theta\_t - \alpha \nabla\_\theta (-J(\theta))\;;$$
  which is exactly what you have in the book.
  
  
  ---
  
  