Answers.Body: <p>I would recommend not trying to think of this in relation to supervised learning.</p>
  <p>The policy <span class="math-container">$\pi(\cdot; \theta)$</span> is simply a function that is parameterised by <span class="math-container">$\theta$</span>. If we take a <span class="math-container">$\log$</span> of this function, it is still just a function. We want to take the (partial) derivative(s) of this function with respect to the parameters so that we can perform a gradient ascent step on the parameters.</p>
  <p>A simple example can be shown by letting <span class="math-container">$\pi(a; \alpha, \beta) = \exp(\alpha + \beta a)$</span>. In the policy gradient theorem we must first take a log of the policy which would give us <span class="math-container">$\log(\pi(a; \alpha, \beta)) = \alpha + \beta a$</span>, and the partial derivatives wrt to the parameters are <span class="math-container">$\nabla_\alpha \log(\pi(a; \alpha, \beta)) = 1$</span> and <span class="math-container">$\nabla_\beta \log(\pi(a; \alpha, \beta)) = a$</span>. We can then use these partial derivatives to perform a gradient ascent update in the direction of the gradient of our objective (the value function, which is of course what we want to maximise) for <span class="math-container">$\alpha$</span> and <span class="math-container">$\beta$</span> for a given return <span class="math-container">$G_t$</span> and action <span class="math-container">$a_t$</span> by
  <span class="math-container">\begin{equation}
  \alpha' = \alpha + G_t \times \nabla_\alpha \log(\pi(a_t; \alpha, \beta)) = \alpha + G_t \times 1 \; \\
  \beta' = \beta + G_t \times \nabla_\beta \log(\pi(a_t; \alpha, \beta)) = \beta + G_t a_t\;.
  \end{equation}</span></p>
  <p>In practice, however, you're likely to need a much more complex function for the policy, typically a neural network of some description. However, everything translates to these more complex functions, you're just going to have many more partial derivatives to calculate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T09:30:20.157Z
Answers.Text: this is perfect - thank you very much! i was having trouble relating the policy itself to still being a differentiable function. thank you for the very clear explanation
Answers.UserDisplayName: user9317212
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-07T20:56:15.330Z
Answers.LastActivityDate: 2023-07-02T00:08:58.433Z
Answers.LastEditDate: 2023-07-02T00:08:58.433Z
Answers.UserDisplayName: David
Answers.Body: <p>Consider a function <span class="math-container">$f(x)$</span> where <span class="math-container">$x$</span> is a random variable, whose distribution depends on <span class="math-container">$\theta$</span>. The objective is to minimize
  <span class="math-container">\begin{align*} 
  \mathbb{E}_x[f(x)] = \int_x f(x) \pi(x, \theta) dx
  \end{align*}</span>
  where <span class="math-container">$\pi(x, \theta)$</span> is the probability density of <span class="math-container">$x$</span> given the parameters <span class="math-container">$\theta$</span> (to be formal, you should use dummy variables in place of x). The gradient is then
  <span class="math-container">\begin{align*} 
  \nabla_{\theta} \mathbb{E}_x[f(x)] = \int_x f(x) \nabla_{\theta}\pi(x, \theta) dx \label{1}\tag{1}
  \end{align*}</span>
  Eq. (1) is essentially the policy gradient. I think when written like Eq. (1), it's clear to see what the gradient actually means.</p>
  <p>To see why this is the policy gradient, first we have
  <span class="math-container">\begin{align*} 
  \int_x f(x) \nabla_{\theta}\pi(x, \theta) dx = \int_x f(x) \nabla_{\theta}\pi(x, \theta) \dfrac{\pi(x, \theta)}{\pi(x, \theta)}dx = \mathbb{E}_x[ f(x) \nabla_{\theta}\log \pi(x, \theta)].
  \end{align*}</span>
  Now interpret <span class="math-container">$x$</span> as the actions, and interpret <span class="math-container">$f(x)$</span> to be the expected return. I omit the full details of this, but hopefully you get the idea.</p>
  <p><strong>The policy gradient is simply the gradient of the expected returns</strong>, with respect to the parameters of the action distribution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T09:30:41.057Z
Answers.Text: thank you! i upvote both answers for their detail and clear explanations
Answers.UserDisplayName: user9317212
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T08:45:48.637Z
Answers.LastActivityDate: 2021-12-08T08:45:48.637Z
Answers.UserDisplayName: Taw
Body: <p>I am getting tripped up slightly by how specifically the gradient is calculated in policy gradient methods (just the intuitive understanding of it).</p>
  <p><a href="https://math.stackexchange.com/q/2845971/168764">This Math Stack Exchange post</a> is close, but I'm still a little confused.</p>
  <p>In standard supervised learning, the partial derivatives can be acquired specifically because we want to learn about the derivative of the cost with respect to input parameters and then adjust in the direction of minimising this error.</p>
  <p>Policy gradients are the opposite, as we want to maximise the likelihood of taking good actions. However, I don't understand what we are getting partial derivatives with respect to - in other words, what is the 'equivalent' of the cost function, specifically for <span class="math-container">$\nabla_\theta \log\pi_\theta$</span>?</p>
  
Tags: <reinforcement-learning><objective-functions><policy-gradients><gradient>
Title: What specifically is the gradient of the log of the probability in policy gradient methods?
UserDisplayName: user9317212
ThreadText: # What specifically is the gradient of the log of the probability in policy gradient methods?
  
  (Asked by: username_0 on 2021-12-07)
  
  I am getting tripped up slightly by how specifically the gradient is calculated in policy gradient methods (just the intuitive understanding of it).
  
  [This Math Stack Exchange post](https://math.stackexchange.com/q/2845971/168764) is close, but I'm still a little confused.
  
  In standard supervised learning, the partial derivatives can be acquired specifically because we want to learn about the derivative of the cost with respect to input parameters and then adjust in the direction of minimising this error.
  
  Policy gradients are the opposite, as we want to maximise the likelihood of taking good actions. However, I don't understand what we are getting partial derivatives with respect to - in other words, what is the 'equivalent' of the cost function, specifically for $\nabla\_\theta \log\pi\_\theta$?
  
  **Tags:** reinforcement-learning, objective-functions, policy-gradients, gradient
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-12-07. Score: 2 | Accepted answer
  
  I would recommend not trying to think of this in relation to supervised learning.
  
  The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by $\theta$. If we take a $\log$ of this function, it is still just a function. We want to take the (partial) derivative(s) of this function with respect to the parameters so that we can perform a gradient ascent step on the parameters.
  
  A simple example can be shown by letting $\pi(a; \alpha, \beta) = \exp(\alpha + \beta a)$. In the policy gradient theorem we must first take a log of the policy which would give us $\log(\pi(a; \alpha, \beta)) = \alpha + \beta a$, and the partial derivatives wrt to the parameters are $\nabla\_\alpha \log(\pi(a; \alpha, \beta)) = 1$ and $\nabla\_\beta \log(\pi(a; \alpha, \beta)) = a$. We can then use these partial derivatives to perform a gradient ascent update in the direction of the gradient of our objective (the value function, which is of course what we want to maximise) for $\alpha$ and $\beta$ for a given return $G\_t$ and action $a\_t$ by
  \begin{equation}
  \alpha' = \alpha + G\_t \times \nabla\_\alpha \log(\pi(a\_t; \alpha, \beta)) = \alpha + G\_t \times 1 \; \\
  \beta' = \beta + G\_t \times \nabla\_\beta \log(\pi(a\_t; \alpha, \beta)) = \beta + G\_t a\_t\;.
  \end{equation}
  
  In practice, however, you're likely to need a much more complex function for the policy, typically a neural network of some description. However, everything translates to these more complex functions, you're just going to have many more partial derivatives to calculate.
  
  > **Comments:**
  > **username_0**: this is perfect - thank you very much! i was having trouble relating the policy itself to still being a differentiable function. thank you for the very clear explanation  
  
  ---
  
  ### Answer by username_2 on 2021-12-08. Score: 3
  
  Consider a function $f(x)$ where $x$ is a random variable, whose distribution depends on $\theta$. The objective is to minimize
  \begin{align\*}
  \mathbb{E}\_x[f(x)] = \int\_x f(x) \pi(x, \theta) dx
  \end{align\*}
  where $\pi(x, \theta)$ is the probability density of $x$ given the parameters $\theta$ (to be formal, you should use dummy variables in place of x). The gradient is then
  \begin{align\*}
  \nabla\_{\theta} \mathbb{E}\_x[f(x)] = \int\_x f(x) \nabla\_{\theta}\pi(x, \theta) dx \label{1}\tag{1}
  \end{align\*}
  Eq. (1) is essentially the policy gradient. I think when written like Eq. (1), it's clear to see what the gradient actually means.
  
  To see why this is the policy gradient, first we have
  \begin{align\*}
  \int\_x f(x) \nabla\_{\theta}\pi(x, \theta) dx = \int\_x f(x) \nabla\_{\theta}\pi(x, \theta) \dfrac{\pi(x, \theta)}{\pi(x, \theta)}dx = \mathbb{E}\_x[ f(x) \nabla\_{\theta}\log \pi(x, \theta)].
  \end{align\*}
  Now interpret $x$ as the actions, and interpret $f(x)$ to be the expected return. I omit the full details of this, but hopefully you get the idea.
  
  **The policy gradient is simply the gradient of the expected returns**, with respect to the parameters of the action distribution.
  
  > **Comments:**
  > **username_0**: thank you! i upvote both answers for their detail and clear explanations  
  
  ---
  
  