Answers.Body: <p>A practical and efficient method to compute the gradient involves leveraging stochastic gradient descent (SGD). This implementation aligns with the training approach commonly used in Variational Autoencoders (VAEs). The key idea is to estimate the gradient of the algorithm by using a mini-batch of size M to approximate the loss over the entire dataset size N. Specifically, the loss function is expressed as:
  <span class="math-container">$\mathcal{L}(D) \approx -\frac{N}{M}\mathcal{L}(\mathbf{X}^M) = -\frac{N}{ML} \sum_{j=1}^{N} \sum_{i=1}^{L} \log p_{\theta}(x_{j} | z) \\
  \mathcal{L}(D) \approx -\frac{N}{M}\mathcal{L}(\mathbf{X}^M) = -\frac{N}{M} \sum_{j=1}^{N} \log p_{\theta}(x_{j} | z)$</span></p>
  <p>By increasing the mini-batch size sufficiently, it becomes feasible to use only one sample for the gradient estimation. This approach is referred to as the single-sample Monte Carlo method.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-19T03:23:53.353Z
Answers.LastActivityDate: 2024-12-18T11:50:00.577Z
Answers.LastEditDate: 2024-12-18T11:50:00.577Z
Answers.UserDisplayName: Đức Cường
Answers.Body: <p>About the first question, you are right. The <span class="math-container">$i$</span> denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose <span class="math-container">$N=1$</span>. </p>
  
  <p>About the second question, the Policy Gradient theorem only tells you what is the gradient up to a constant, so basically any constant is irrelevant. Now, even if you do know the constant, you are going to multiply the gradients by an arbitrary learning rate <span class="math-container">$\alpha$</span>. So, you can think that the factor <span class="math-container">$\frac{1}{N}$</span> is actually already considered "inside" <span class="math-container">$\alpha$</span>.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-08T21:23:27.677Z
Answers.Text: Could you please detail on the second point. I don't really understand how the policy gradient theorem is related to averaging up the batch
Answers.UserDisplayName: d56
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-08T22:23:44.637Z
Answers.Text: The PG theorem tells you that the gradient  $\nabla J$ is proportional to the expected value of $G\sum_t\nabla\log\pi(a_t|s_t)$. So, your actual gradient is $C\mathbb E\left[G\sum_t\nabla\log\pi(a_t|s_t)\right]$, where $C$ is some constant. If you perform an SGD step with learning rate $\eta$, then your new parameters will be $\theta\leftarrow\theta+\eta\nabla J \approx\theta+\frac{C\eta}{N}\sum_iG_i\sum_t\nabla\log\pi(a_t^i|s_t^i) $. If you define $\alpha=\frac{C\eta}{N}$, you will obtain Levine's version. Since $\eta$ is arbitrary, you can see that $C$ or $\frac{1}{N}$ are irrelevant.
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-08T22:27:20.060Z
Answers.Text: So, to answer your question, the fact that we usually don't know $C$ and that the learning rate $\eta$ is arbitrary makes irrelevant if you decide to consider the sum or the mean over the trajectories, that is, both are valid approximations of your gradient.
Answers.UserDisplayName: Diego Gomez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-08T02:47:41.710Z
Answers.LastActivityDate: 2020-01-08T03:02:51.393Z
Answers.LastEditDate: 2020-01-08T03:02:51.393Z
Answers.UserDisplayName: Diego Gomez
Body: <p>I followed the videos/slides of <a href="http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-5.pdf" rel="nofollow noreferrer">Berkley RL course</a>, but now I am a bit confused when implementing it. Please see the picture below. </p>
  
  <p><a href="https://i.sstatic.net/pfKil.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pfKil.png" alt="lide 9"></a></p>
  
  <p>In particular, what does <span class="math-container">$i$</span> represent in the REINFORCE algorithm? If <span class="math-container">$\tau^i$</span> is the trajectory for the whole episode <span class="math-container">$i$</span>, then why don't we average across the episodes <span class="math-container">$\frac{1}{N}$</span>, which approximates the gradient of the objective function? Instead, it is a sum over the <span class="math-container">$i$</span>. So, do we update the gradients per episode or have batches of episodes to update it? 
  When I compare the algorithm to Sutton's book as shown below, I see that there we update the gradients per episode. </p>
  
  <p><a href="https://i.sstatic.net/8Flg4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8Flg4.png" alt="enter image description here"></a></p>
  
  <p>But wouldn't it then contradict the derivation on the Levine's slide that the gradient of the objective function <span class="math-container">$J$</span> is the expectation (therefore sampling) of the gradients of the logs? </p>
  
  <p>Secondly, why do we have a cumulative sum of the returns over <span class="math-container">$T$</span> in Sutton's version but do not do it in Levine's (instead, all returns are summed together)   </p>
  
Tags: <reinforcement-learning><comparison><policy-gradients><reinforce>
Title: What is the difference between Sutton's and Levine's REINFORCE algorithm?
UserDisplayName: d56
ThreadText: # What is the difference between Sutton's and Levine's REINFORCE algorithm?
  
  (Asked by: username_0 on 2020-01-07)
  
  I followed the videos/slides of [Berkley RL course](http://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-5.pdf), but now I am a bit confused when implementing it. Please see the picture below.
  
  [![lide 9](https://i.sstatic.net/pfKil.png)](https://i.sstatic.net/pfKil.png)
  
  In particular, what does $i$ represent in the REINFORCE algorithm? If $\tau^i$ is the trajectory for the whole episode $i$, then why don't we average across the episodes $\frac{1}{N}$, which approximates the gradient of the objective function? Instead, it is a sum over the $i$. So, do we update the gradients per episode or have batches of episodes to update it?
  When I compare the algorithm to Sutton's book as shown below, I see that there we update the gradients per episode.
  
  [![enter image description here](https://i.sstatic.net/8Flg4.png)](https://i.sstatic.net/8Flg4.png)
  
  But wouldn't it then contradict the derivation on the Levine's slide that the gradient of the objective function $J$ is the expectation (therefore sampling) of the gradients of the logs?
  
  Secondly, why do we have a cumulative sum of the returns over $T$ in Sutton's version but do not do it in Levine's (instead, all returns are summed together)
  
  **Tags:** reinforcement-learning, comparison, policy-gradients, reinforce
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-01-08. Score: 2 | Accepted answer
  
  About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$.
  
  About the second question, the Policy Gradient theorem only tells you what is the gradient up to a constant, so basically any constant is irrelevant. Now, even if you do know the constant, you are going to multiply the gradients by an arbitrary learning rate $\alpha$. So, you can think that the factor $\frac{1}{N}$ is actually already considered "inside" $\alpha$.
  
  > **Comments:**
  > **username_0**: Could you please detail on the second point. I don't really understand how the policy gradient theorem is related to averaging up the batch  
  > **username_2**: The PG theorem tells you that the gradient  $\nabla J$ is proportional to the expected value of $G\sum_t\nabla\log\pi(a_t|s_t)$. So, your actual gradient is $C\mathbb E\left[G\sum_t\nabla\log\pi(a_t|s_t)\right]$, where $C$ is some constant. If you perform an SGD step with learning rate $\eta$, then your new parameters will be $\theta\leftarrow\theta+\eta\nabla J \approx\theta+\frac{C\eta}{N}\sum_iG_i\sum_t\nabla\log\pi(a_t^i|s_t^i) $. If you define $\alpha=\frac{C\eta}{N}$, you will obtain Levine's version. Since $\eta$ is arbitrary, you can see that $C$ or $\frac{1}{N}$ are irrelevant.  
  > **username_2**: So, to answer your question, the fact that we usually don't know $C$ and that the learning rate $\eta$ is arbitrary makes irrelevant if you decide to consider the sum or the mean over the trajectories, that is, both are valid approximations of your gradient.  
  
  ---
  
  ### Answer by username_1 on 2024-01-19. Score: 0
  
  A practical and efficient method to compute the gradient involves leveraging stochastic gradient descent (SGD). This implementation aligns with the training approach commonly used in Variational Autoencoders (VAEs). The key idea is to estimate the gradient of the algorithm by using a mini-batch of size M to approximate the loss over the entire dataset size N. Specifically, the loss function is expressed as:
  $\mathcal{L}(D) \approx -\frac{N}{M}\mathcal{L}(\mathbf{X}^M) = -\frac{N}{ML} \sum\_{j=1}^{N} \sum\_{i=1}^{L} \log p\_{\theta}(x\_{j} | z) \\
  \mathcal{L}(D) \approx -\frac{N}{M}\mathcal{L}(\mathbf{X}^M) = -\frac{N}{M} \sum\_{j=1}^{N} \log p\_{\theta}(x\_{j} | z)$
  
  By increasing the mini-batch size sufficiently, it becomes feasible to use only one sample for the gradient estimation. This approach is referred to as the single-sample Monte Carlo method.
  
  
  ---
  
  