Answers.Body: <p>In REINFORCE, if you generated several episodes, and calculated the gradient over all transitions over all episodes, this would reduce the variance of the gradient compared to regular REINFORCE where we sample one episode at a time. You might know that when estimating the sample mean of a population, the variance decreases like <span class="math-container">$1/n$</span> where <span class="math-container">$n$</span> is the sample size. That's true here, for exactly the same reason: if you generated <span class="math-container">$n$</span> episodes per REINFORCE gradient, the variance will be <span class="math-container">$1/n$</span> what it is in normal REINFORCE.</p>
  <p>If we choose some <span class="math-container">$n$</span> and also multiply the learning rate by <span class="math-container">$n$</span>, we would expect both versions of REINFORCE to perform about the same in terms of average reward vs wall time and average reward vs number of episodes. But the one with higher <span class="math-container">$n$</span> does less gradient updates. In practice, you might be able to tune <span class="math-container">$n$</span> as a hyperparameter, but really you need to be using a better algorithm than REINFORCE if you care about performance at all.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-03T17:37:45.467Z
Answers.Text: Just adding that the contrast you made against PPO in the other answer is helpful to establish the last point (better using a diff algorithm) and provides some intuitions for the effect on REINFORCE as well (in case other readers see this).
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-03T14:38:51.647Z
Answers.LastActivityDate: 2022-10-03T14:38:51.647Z
Answers.UserDisplayName: Taw
Answers.Body: <p>Yes I see in the repo you link, in reinforce.py they only perform a gradient update once every episode. It sounds like what you're asking about is the difference between that reinforce style, and the more popular (and also more efficient) PPO type style. In that latter way, we have something like
  <span class="math-container">\begin{align*} 
  &amp; \text{ for each iteration }: \\
  &amp; \qquad \text{ for t in range(size_training_set)}: \\
  &amp; \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1}; \text{ save transition to memory} \\
  &amp; \qquad \text{ for m epochs}: \\
  &amp; \qquad \qquad \text{ calculate advantages} \\
  &amp; \qquad \qquad \text{ for k mini-batches}: \\
  &amp; \qquad \qquad \qquad \text{make mini batch from training_set}\text{ and do policy gradient update }
  \end{align*}</span>
  There are some other details such as importance sampling, so I would recommend you can try another repo's code first.</p>
  <p>The advantage of the PPO way is that we spend more time training on mini-batches and less time sampling the environment (which is slower), we can use each transition in multiple mini-batches, and we can generate more varied data to train on. Grouping together transitions from different times in different episodes might help remove harmful correlations. Also, theoretically the batch size shouldn't really matter, but in practice it's important, and we can't even tune that with the reinforce way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-02T20:56:26.903Z
Answers.Text: This is a very helpful read to get more information on PPO and shows what additional it offers as contrast, but I'm afraid I was asking explicitly in the REINFORCE like context. In my case, I imagine batches as sets of episodes (or trajectories). Anecdotally I've so far found this to not "improve" training but it does decrease the variance ever so slightly while requiring more iterations. Perhaps the benefit in my case is reduced variance (stabilization) and reduced number of weight updates (possible speed, but it's a bit slower on raw epoch count), though it's unclear.
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-03T00:54:47.620Z
Answers.Text: Ok I probably didn't understand your question then. You're considering reinforce that generates, say, 4 episodes, and then does the "batch" over all transitions over all 4 episodes? That seems like a really weird thing to do to me.
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-03T14:21:35.190Z
Answers.Text: Correct. What I would be interested in learning is why that might be weird, if there are better alternatives (that still are ~ within the REINFORCE territory), and better conceptual understanding of it all.
Answers.UserDisplayName: Josh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-02T01:48:47.627Z
Answers.LastActivityDate: 2022-10-02T01:48:47.627Z
Answers.UserDisplayName: Taw
Body: <p>I'm following along with PyTorch's example implementations (<a href="https://github.com/pytorch/examples/tree/main/reinforcement_learning" rel="nofollow noreferrer">found here</a>) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don't use batches. This leads me to ask, are batch updates of the network actually useful in this context?</p>
  <p>Adding on, in my particular environment there's not a real meaningful cutoff for episodes as it's really set up for a sort of continuous play. As such, any n-length trajectory + rewards I collect is just as valid as another. For that reason, it would seem to mean that  a longer episode/trajectory would serve the same purpose batches tend to in network updating.</p>
  <p>Is it expected then that batches are not particularly worthwhile in the REINFORCE context, or is this just coincidence of the implementation I'm using? And is that answer amended if there are no meaningful episode cutoffs?</p>
  
Tags: <reinforcement-learning><deep-rl><reinforce><batch-learning>
Title: Are batches useful for REINFORCE without strong episode cutoffs?
UserDisplayName: Josh
ThreadText: # Are batches useful for REINFORCE without strong episode cutoffs?
  
  (Asked by: username_0 on 2022-09-20)
  
  I'm following along with PyTorch's example implementations ([found here](https://github.com/pytorch/examples/tree/main/reinforcement_learning)) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don't use batches. This leads me to ask, are batch updates of the network actually useful in this context?
  
  Adding on, in my particular environment there's not a real meaningful cutoff for episodes as it's really set up for a sort of continuous play. As such, any n-length trajectory + rewards I collect is just as valid as another. For that reason, it would seem to mean that a longer episode/trajectory would serve the same purpose batches tend to in network updating.
  
  Is it expected then that batches are not particularly worthwhile in the REINFORCE context, or is this just coincidence of the implementation I'm using? And is that answer amended if there are no meaningful episode cutoffs?
  
  **Tags:** reinforcement-learning, deep-rl, reinforce, batch-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-10-03. Score: 2 | Accepted answer
  
  In REINFORCE, if you generated several episodes, and calculated the gradient over all transitions over all episodes, this would reduce the variance of the gradient compared to regular REINFORCE where we sample one episode at a time. You might know that when estimating the sample mean of a population, the variance decreases like $1/n$ where $n$ is the sample size. That's true here, for exactly the same reason: if you generated $n$ episodes per REINFORCE gradient, the variance will be $1/n$ what it is in normal REINFORCE.
  
  If we choose some $n$ and also multiply the learning rate by $n$, we would expect both versions of REINFORCE to perform about the same in terms of average reward vs wall time and average reward vs number of episodes. But the one with higher $n$ does less gradient updates. In practice, you might be able to tune $n$ as a hyperparameter, but really you need to be using a better algorithm than REINFORCE if you care about performance at all.
  
  > **Comments:**
  > **username_0**: Just adding that the contrast you made against PPO in the other answer is helpful to establish the last point (better using a diff algorithm) and provides some intuitions for the effect on REINFORCE as well (in case other readers see this).  
  
  ---
  
  ### Answer by username_1 on 2022-10-02. Score: 0
  
  Yes I see in the repo you link, in reinforce.py they only perform a gradient update once every episode. It sounds like what you're asking about is the difference between that reinforce style, and the more popular (and also more efficient) PPO type style. In that latter way, we have something like
  \begin{align\*}
  & \text{ for each iteration }: \\
  & \qquad \text{ for t in range(size\_training\_set)}: \\
  & \qquad \qquad \text{sample } a\_t; \text{ get reward } r\_t \text{ and next state } s\_{t+1}; \text{ save transition to memory} \\
  & \qquad \text{ for m epochs}: \\
  & \qquad \qquad \text{ calculate advantages} \\
  & \qquad \qquad \text{ for k mini-batches}: \\
  & \qquad \qquad \qquad \text{make mini batch from training\_set}\text{ and do policy gradient update }
  \end{align\*}
  There are some other details such as importance sampling, so I would recommend you can try another repo's code first.
  
  The advantage of the PPO way is that we spend more time training on mini-batches and less time sampling the environment (which is slower), we can use each transition in multiple mini-batches, and we can generate more varied data to train on. Grouping together transitions from different times in different episodes might help remove harmful correlations. Also, theoretically the batch size shouldn't really matter, but in practice it's important, and we can't even tune that with the reinforce way.
  
  > **Comments:**
  > **username_0**: This is a very helpful read to get more information on PPO and shows what additional it offers as contrast, but I'm afraid I was asking explicitly in the REINFORCE like context. In my case, I imagine batches as sets of episodes (or trajectories). Anecdotally I've so far found this to not "improve" training but it does decrease the variance ever so slightly while requiring more iterations. Perhaps the benefit in my case is reduced variance (stabilization) and reduced number of weight updates (possible speed, but it's a bit slower on raw epoch count), though it's unclear.  
  > **username_1**: Ok I probably didn't understand your question then. You're considering reinforce that generates, say, 4 episodes, and then does the "batch" over all transitions over all 4 episodes? That seems like a really weird thing to do to me.  
  > **username_0**: Correct. What I would be interested in learning is why that might be weird, if there are better alternatives (that still are ~ within the REINFORCE territory), and better conceptual understanding of it all.  
  
  ---
  
  