Answers.Body: <p>We don't need multiple environments.
  On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest policy to collect multiple samples, even over multiple epochs, before updating the weights. This update can be a batch update.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-12T23:01:05.337Z
Answers.LastActivityDate: 2020-10-12T23:01:05.337Z
Answers.UserDisplayName: Tom Dörr
Body: <p>When using an on-policy method in reinforcement learning, like <em>advantage actor-critic</em>, you shouldn't use old data from an <em>experience buffer</em>, since a new policy requires new data. Does this mean that to apply batching to an on-policy method you have to have multiple parallel environments?</p>
  <p>As an extension of this, if only one environment is available when using on-policy methods, does that mean batching isn't possible? Doesn't that limit the power of such algorithms in certain cases?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-12T12:56:32.920Z
Comments.Text: [Here](https://ai.stackexchange.com/q/21109/2444), [here](https://ai.stackexchange.com/q/20189/2444) and [here](https://ai.stackexchange.com/q/20871/2444) are 3 related questions.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><environment><policies><experience-replay><on-policy-methods>
Title: Do we need multiple parallel environments to train in batches an on-policy algorithm?
UserDisplayName: Daniel
ThreadText: # Do we need multiple parallel environments to train in batches an on-policy algorithm?
  
  (Asked by: username_0 on 2020-07-02)
  
  When using an on-policy method in reinforcement learning, like *advantage actor-critic*, you shouldn't use old data from an *experience buffer*, since a new policy requires new data. Does this mean that to apply batching to an on-policy method you have to have multiple parallel environments?
  
  As an extension of this, if only one environment is available when using on-policy methods, does that mean batching isn't possible? Doesn't that limit the power of such algorithms in certain cases?
  
  **Tags:** reinforcement-learning, environment, policies, experience-replay, on-policy-methods
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/21109/2444), [here](https://ai.stackexchange.com/q/20189/2444) and [here](https://ai.stackexchange.com/q/20871/2444) are 3 related questions.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-10-12. Score: 1
  
  We don't need multiple environments.
  On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest policy to collect multiple samples, even over multiple epochs, before updating the weights. This update can be a batch update.
  
  
  ---
  
  