Body: <p>I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : <a href="https://towardsdatascience.com/policy-gradient-methods-104c783251e0" rel="nofollow noreferrer">https://towardsdatascience.com/policy-gradient-methods-104c783251e0</a> that the gradient of the objective function is calculated as follows:</p>
  <p><a href="https://i.sstatic.net/rEpcw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rEpcw.png" alt="enter image description here" /></a></p>
  <p>From what I understand <span class="math-container">$\sum_{t=0}^{H}\nabla_{\theta}\log{\pi_{\theta}(a_{t}|s_{t})}$</span> is the sum through the entire trajectory and <span class="math-container">$\pi_{\theta}(a_{t}|s_{t})$</span> is the policy of the agent at time step <span class="math-container">$t$</span>. However in Suton's book the gradient objective is defined differently.</p>
  <p><a href="https://i.sstatic.net/C987c.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/C987c.png" alt="enter image description here" /></a></p>
  <p>There is only <span class="math-container">$\nabla \ln{\pi(A_t | S_t)}$</span> at time step <span class="math-container">$t$</span> and no sum of all time steps. So does the algorithm not consider the policy for the whole trajectory when updating? Only a single-step policy?</p>
  <p>Furthermore, there is <span class="math-container">$\gamma^{t}$</span> (discounted reward) term in the latter and not the former. What is the reason for that?</p>
  <p>Hopefully, someone can help me clarify this.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T22:16:21.363Z
Comments.Text: the site you learn from accumulates the gradient at each action taken in the episode via the sum and performs a single gradient ascent step for all these actions (this is analogous to taking the loss over a batch before performing a gradient update in supervised learning), whereas Sutton and Barto perform a gradient ascent step for every action taken (analogous to performing a gradient update for every single data point individually in supervised learning).
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-04T08:54:12.043Z
Comments.Text: @norbertk thanks, I missed the bit of the question about the second discount term. 
  
  OP, essentially the second discount term is needed because you also need to discount your state distribution. This is rarely done in practice but it is theoretically incorrect to omit it, and
  [this](https://arxiv.org/abs/1906.07073) paper shows without the second discount factor the gradient is not actually a gradient at all.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-06T11:10:06.893Z
Comments.Text: Hello. Welcome to AI SE. Rather than writing "Help with understanding the REINFORCE algorithm", please, just put your **specific** question in the title.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><math><reinforce>
Title: REINFORCE differentiation on sum or single value?
UserDisplayName: Kronic
ThreadText: # REINFORCE differentiation on sum or single value?
  
  (Asked by: username_0 on 2021-09-03)
  
  I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : <https://towardsdatascience.com/policy-gradient-methods-104c783251e0> that the gradient of the objective function is calculated as follows:
  
  [![enter image description here](https://i.sstatic.net/rEpcw.png)](https://i.sstatic.net/rEpcw.png)
  
  From what I understand $\sum\_{t=0}^{H}\nabla\_{\theta}\log{\pi\_{\theta}(a\_{t}|s\_{t})}$ is the sum through the entire trajectory and $\pi\_{\theta}(a\_{t}|s\_{t})$ is the policy of the agent at time step $t$. However in Suton's book the gradient objective is defined differently.
  
  [![enter image description here](https://i.sstatic.net/C987c.png)](https://i.sstatic.net/C987c.png)
  
  There is only $\nabla \ln{\pi(A\_t | S\_t)}$ at time step $t$ and no sum of all time steps. So does the algorithm not consider the policy for the whole trajectory when updating? Only a single-step policy?
  
  Furthermore, there is $\gamma^{t}$ (discounted reward) term in the latter and not the former. What is the reason for that?
  
  Hopefully, someone can help me clarify this.
  
  **Tags:** reinforcement-learning, math, reinforce
  
  > **Comments:**
  > **username_1**: the site you learn from accumulates the gradient at each action taken in the episode via the sum and performs a single gradient ascent step for all these actions (this is analogous to taking the loss over a batch before performing a gradient update in supervised learning), whereas Sutton and Barto perform a gradient ascent step for every action taken (analogous to performing a gradient update for every single data point individually in supervised learning).  
  > **username_1**: @norbertk thanks, I missed the bit of the question about the second discount term.   OP, essentially the second discount term is needed because you also need to discount your state distribution. This is rarely done in practice but it is theoretically incorrect to omit it, and [this](https://arxiv.org/abs/1906.07073) paper shows without the second discount factor the gradient is not actually a gradient at all.  
  > **username_2**: Hello. Welcome to AI SE. Rather than writing "Help with understanding the REINFORCE algorithm", please, just put your **specific** question in the title.  
  
  ---
  
  ## 0 answers
  
  