Answers.Body: <p>The value of the objective depends on policy (probabilities of taking an action). Intuitively speaking, better actions lead to better returns and by &quot;pushing up&quot; the probabilities (log or not same thing since log is monotonically increasing function) of those actions you're making sure you're getting better returns and increasing the value of your objective.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-09T15:10:10.387Z
Answers.LastActivityDate: 2021-07-09T15:10:10.387Z
Answers.UserDisplayName: Brale
Body: <p>I'm currently going through the <a href="https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html" rel="nofollow noreferrer">OpenAI's spinning up introduction course to reinforcement learning</a>. On one of the final sections, they derive an expression for the gradient of the undiscounted return with respect to the policy weights:</p>
  <p><span class="math-container">$$\nabla_{\theta} J\left(\pi_{\theta}\right)=\underset{\tau \sim \pi_{\theta}}{\mathrm{E}}\left[\sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) R(\tau)\right]$$</span></p>
  <p>Then they give the following explanation:</p>
  <blockquote>
  <p>Taking a step with this gradient pushes up the log-probabilities of each action in proportion to <span class="math-container">$R(\tau$</span>).</p>
  </blockquote>
  <p>My question is: How does this expression mathematically reflect the fact that this gradient will push up the log probabilities of the actions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-09T12:58:23.840Z
Comments.Text: The phrase you quote is literally an English reading of the notation in your image. Would an answer that explained what $\nabla_{\theta}$, $\sum$, $\pi_{\theta}$  etc all mean help you in any way? Or is that part clear to you, and you are looking for something more intuitive?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-09T14:47:34.093Z
Comments.Text: I am already familiar with the notations. What I'm struggling to understand is how does this mathematical expression translate into the english explanation given. As I understand it, this gradient tell us how to update the weights so that we increase the return J and we need the grad log prob of the actions to compute it. But why does this also pushes up the log probabilities at the same time? Maybe something more intuitive could help. Thanks for your time.
Comments.UserDisplayName: Samuel Beaussant
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-06-21T10:08:38.350Z
Comments.Text: Logarithm of Pi has the same monotonicity of policy Pi and its peaks and valleys are at the same loci as Pi's. Given some rewards for a specific trajectory, we assign the heavier weights to the actions as peaks in policy Pi corresponding to the highest rewards in order to maximize the total rewards. This is done by Gradient Ascent Algorithm for policy Pi(Remember Back Propagation). PS., we introduce Logarithm so that we can apply Sampling Method to estimate the reward for a trajectory.
Comments.UserDisplayName: Kuo
Tags: <reinforcement-learning><deep-rl><math><policy-gradients>
Title: How to interpret the policy gradient expression in reinforcement learning?
UserDisplayName: Samuel Beaussant
ThreadText: # How to interpret the policy gradient expression in reinforcement learning?
  
  (Asked by: username_0 on 2021-07-09)
  
  I'm currently going through the [OpenAI's spinning up introduction course to reinforcement learning](https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html). On one of the final sections, they derive an expression for the gradient of the undiscounted return with respect to the policy weights:
  
  $$\nabla\_{\theta} J\left(\pi\_{\theta}\right)=\underset{\tau \sim \pi\_{\theta}}{\mathrm{E}}\left[\sum\_{t=0}^{T} \nabla\_{\theta} \log \pi\_{\theta}\left(a\_{t} \mid s\_{t}\right) R(\tau)\right]$$
  
  Then they give the following explanation:
  
  > Taking a step with this gradient pushes up the log-probabilities of each action in proportion to $R(\tau$).
  
  My question is: How does this expression mathematically reflect the fact that this gradient will push up the log probabilities of the actions?
  
  **Tags:** reinforcement-learning, deep-rl, math, policy-gradients
  
  > **Comments:**
  > **username_1**: The phrase you quote is literally an English reading of the notation in your image. Would an answer that explained what $\nabla_{\theta}$, $\sum$, $\pi_{\theta}$  etc all mean help you in any way? Or is that part clear to you, and you are looking for something more intuitive?  
  > **username_0**: I am already familiar with the notations. What I'm struggling to understand is how does this mathematical expression translate into the english explanation given. As I understand it, this gradient tell us how to update the weights so that we increase the return J and we need the grad log prob of the actions to compute it. But why does this also pushes up the log probabilities at the same time? Maybe something more intuitive could help. Thanks for your time.  
  > **username_2**: Logarithm of Pi has the same monotonicity of policy Pi and its peaks and valleys are at the same loci as Pi's. Given some rewards for a specific trajectory, we assign the heavier weights to the actions as peaks in policy Pi corresponding to the highest rewards in order to maximize the total rewards. This is done by Gradient Ascent Algorithm for policy Pi(Remember Back Propagation). PS., we introduce Logarithm so that we can apply Sampling Method to estimate the reward for a trajectory.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-07-09. Score: 1
  
  The value of the objective depends on policy (probabilities of taking an action). Intuitively speaking, better actions lead to better returns and by "pushing up" the probabilities (log or not same thing since log is monotonically increasing function) of those actions you're making sure you're getting better returns and increasing the value of your objective.
  
  
  ---
  
  