Answers.Body: <blockquote>
  <p>Do parallel environments improve the agent's ability to learn or does it not really make a difference?</p>
  </blockquote>
  <p>Yes they can make a difference. There are two ways improvement is seen:</p>
  <ul>
  <li><p>Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with <a href="https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables" rel="nofollow noreferrer">i.i.d. data</a>.</p>
  </li>
  <li><p>Data collection is faster overall, which improves clock time to obtain the same result. This may make better use of other resources too.</p>
  </li>
  </ul>
  <p>Of the two, the first improvement is important for stability, although it can be emulated by running multiple episodes - or restarting from multiple starting points - between batch learning updates.</p>
  <blockquote>
  <p>Specifically, I am using PPO, but I think this applies across the board to other algorithms too.</p>
  </blockquote>
  <p>It does apply to PPO, but the first improvement does not apply across the board. These things need to be true for environment run in paralell to be help with stability:</p>
  <ul>
  <li><p>Using an on-policy method, or where experience replay is not an option.</p>
  </li>
  <li><p>Using a function approximator for policy and/or value function.</p>
  </li>
  </ul>
  <p>A lot of policy gradient methods match this, including PPO, A3C, REINFORCE. However, for an off-policy method like DQN, the main benefit will be faster data collection.</p>
  <p>These effects are discussed in sections 1 and 4 of the paper <a href="https://arxiv.org/pdf/1602.01783.pdf" rel="nofollow noreferrer">Asynchronous Methods for Deep Reinforcement Learning</a> which introduced A3C - thanks to DeepQZero for that reference.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-23T22:47:22.883Z
Answers.Text: Why only for on-policy and why for no exp.replay?
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-24T06:42:43.143Z
Answers.Text: @ddaedalus Experiance replay is only feasible for off-policy, because older records will have been made with a different policy - off-policy can handle that, on-policy cannot. If you have experience replay, you already have the benefit of improved convergence from parallel trajectories, because you already have closer to i.i.d. from sampling the experience replay memory.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-23T16:05:36.177Z
Answers.LastActivityDate: 2021-05-25T17:04:23.143Z
Answers.LastEditDate: 2021-05-25T17:04:23.143Z
Answers.UserDisplayName: Neil Slater
Body: <p>Do parallel environments improve the agent's ability to learn or does it not really make a difference? Specifically, I am using PPO, but I think this applies across the board to other algorithms too.</p>
  
Tags: <reinforcement-learning><deep-rl><proximal-policy-optimization>
Title: What is the effect of parallel environments in reinforcement learning?
UserDisplayName: Dylan Kerler
ThreadText: # What is the effect of parallel environments in reinforcement learning?
  
  (Asked by: username_0 on 2021-05-23)
  
  Do parallel environments improve the agent's ability to learn or does it not really make a difference? Specifically, I am using PPO, but I think this applies across the board to other algorithms too.
  
  **Tags:** reinforcement-learning, deep-rl, proximal-policy-optimization
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-23. Score: 5 | Accepted answer
  
  > Do parallel environments improve the agent's ability to learn or does it not really make a difference?
  
  Yes they can make a difference. There are two ways improvement is seen:
  
  * Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with [i.i.d. data](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables).
  * Data collection is faster overall, which improves clock time to obtain the same result. This may make better use of other resources too.
  
  Of the two, the first improvement is important for stability, although it can be emulated by running multiple episodes - or restarting from multiple starting points - between batch learning updates.
  
  > Specifically, I am using PPO, but I think this applies across the board to other algorithms too.
  
  It does apply to PPO, but the first improvement does not apply across the board. These things need to be true for environment run in paralell to be help with stability:
  
  * Using an on-policy method, or where experience replay is not an option.
  * Using a function approximator for policy and/or value function.
  
  A lot of policy gradient methods match this, including PPO, A3C, REINFORCE. However, for an off-policy method like DQN, the main benefit will be faster data collection.
  
  These effects are discussed in sections 1 and 4 of the paper [Asynchronous Methods for Deep Reinforcement Learning](https://arxiv.org/pdf/1602.01783.pdf) which introduced A3C - thanks to DeepQZero for that reference.
  
  > **Comments:**
  > **username_2**: Why only for on-policy and why for no exp.replay?  
  > **username_1**: @ddaedalus Experiance replay is only feasible for off-policy, because older records will have been made with a different policy - off-policy can handle that, on-policy cannot. If you have experience replay, you already have the benefit of improved convergence from parallel trajectories, because you already have closer to i.i.d. from sampling the experience replay memory.  
  
  ---
  
  