Answers.Body: <p>No, the difference between the gradients will be even greater, as in the second formulation the weights are updated each step, so the gradient is not a cumulative gradient, but a sort of 1-sample-SGD</p>
  <p>In my opinion, the first one is the more accurate, as the following one does not consider the change in parameters (given <span class="math-container">$\theta' = \theta + \nabla_\theta L$</span>, <span class="math-container">$\pi_\theta(A|S) \ne \pi_{\theta'}(A|S)$</span>)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-24T19:21:17.290Z
Answers.LastActivityDate: 2023-09-24T19:21:17.290Z
Answers.UserDisplayName: Alberto
Body: <p>I am a bit confused between two definitions of the Vanilla REINFORCE algorithm.
  The first one is in the following (from this page: <a href="https://stjohngrimbly.com/model-free-RL/" rel="nofollow noreferrer">https://stjohngrimbly.com/model-free-RL/</a>):</p>
  <p><a href="https://i.sstatic.net/xiJJO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/xiJJO.png" alt="First" /></a></p>
  <p>Here, at every step in an episode we calculate the returns with involving the discount factors, such that the final gradient is like the following:
  <span class="math-container">$$\nabla J(\theta)=G_0\nabla_{\theta}log\pi_{\theta}(A_0|S_0) + G_1\nabla_{\theta}log\pi_{\theta}(A_1|S_1) + G_2\nabla_{\theta}log\pi_{\theta}(A_2|S_2) + \dots G_{T-1}\nabla_{\theta}log\pi_{\theta}(A_{T-1}|S_{T-1})$$</span></p>
  <p>However, in Sutton´s book, the definition is like the following:</p>
  <p><a href="https://i.sstatic.net/v1hts.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v1hts.png" alt="Second" /></a></p>
  <p>So, they explicity multiply each gradient term from each time step t, separately with the discount factor <span class="math-container">$\gamma^t$</span>. Assuming they define <span class="math-container">$G_t$</span> similarly like <span class="math-container">$G_t = \sum_{k=t}^{T-1}\gamma^{k-t}r_k$</span>, the obtained gradient will be:</p>
  <p><span class="math-container">$$\nabla J(\theta)=G_0\nabla_{\theta}log\pi_{\theta}(A_0|S_0) + \gamma G_1\nabla_{\theta}log\pi_{\theta}(A_1|S_1) + \gamma^2G_2\nabla_{\theta}log\pi_{\theta}(A_2|S_2) + \dots \gamma^{T-1}G_{T-1}\nabla_{\theta}log\pi_{\theta}(A_{T-1}|S_{T-1})$$</span></p>
  <p>So, the gradient term in total will be different from the first one. Since in Sutton`s version they have used the discount factor reduntantly (it should be in <span class="math-container">$G_t$</span> already), it seems like the first version is the valid one to me, if this has not been done with a specific purpose. My question is, is there a small notational mistake in Sutton version of the algorithm, or am I missing something with the gradient formula here?</p>
  
Tags: <reinforcement-learning><policy-gradients><reinforce>
Title: Discount factors in REINFORCE Algorithm: Difference in two definitions
UserDisplayName: Ufuk Can Bicici
ThreadText: # Discount factors in REINFORCE Algorithm: Difference in two definitions
  
  (Asked by: username_0 on 2023-09-24)
  
  I am a bit confused between two definitions of the Vanilla REINFORCE algorithm.
  The first one is in the following (from this page: <https://stjohngrimbly.com/model-free-RL/>):
  
  [![First](https://i.sstatic.net/xiJJO.png)](https://i.sstatic.net/xiJJO.png)
  
  Here, at every step in an episode we calculate the returns with involving the discount factors, such that the final gradient is like the following:
  $$\nabla J(\theta)=G\_0\nabla\_{\theta}log\pi\_{\theta}(A\_0|S\_0) + G\_1\nabla\_{\theta}log\pi\_{\theta}(A\_1|S\_1) + G\_2\nabla\_{\theta}log\pi\_{\theta}(A\_2|S\_2) + \dots G\_{T-1}\nabla\_{\theta}log\pi\_{\theta}(A\_{T-1}|S\_{T-1})$$
  
  However, in Sutton´s book, the definition is like the following:
  
  [![Second](https://i.sstatic.net/v1hts.png)](https://i.sstatic.net/v1hts.png)
  
  So, they explicity multiply each gradient term from each time step t, separately with the discount factor $\gamma^t$. Assuming they define $G\_t$ similarly like $G\_t = \sum\_{k=t}^{T-1}\gamma^{k-t}r\_k$, the obtained gradient will be:
  
  $$\nabla J(\theta)=G\_0\nabla\_{\theta}log\pi\_{\theta}(A\_0|S\_0) + \gamma G\_1\nabla\_{\theta}log\pi\_{\theta}(A\_1|S\_1) + \gamma^2G\_2\nabla\_{\theta}log\pi\_{\theta}(A\_2|S\_2) + \dots \gamma^{T-1}G\_{T-1}\nabla\_{\theta}log\pi\_{\theta}(A\_{T-1}|S\_{T-1})$$
  
  So, the gradient term in total will be different from the first one. Since in Sutton`s version they have used the discount factor reduntantly (it should be in $G\_t$ already), it seems like the first version is the valid one to me, if this has not been done with a specific purpose. My question is, is there a small notational mistake in Sutton version of the algorithm, or am I missing something with the gradient formula here?
  
  **Tags:** reinforcement-learning, policy-gradients, reinforce
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-24. Score: 1
  
  No, the difference between the gradients will be even greater, as in the second formulation the weights are updated each step, so the gradient is not a cumulative gradient, but a sort of 1-sample-SGD
  
  In my opinion, the first one is the more accurate, as the following one does not consider the change in parameters (given $\theta' = \theta + \nabla\_\theta L$, $\pi\_\theta(A|S) \ne \pi\_{\theta'}(A|S)$)
  
  
  ---
  
  