Answers.Body: <p>OpenAI have a post on that: <a href="https://openai.com/blog/openai-five/" rel="nofollow noreferrer">https://openai.com/blog/openai-five/</a></p>
  
  <p>They use a myriad of <em>rollout workers</em> that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.</p>
  
  <p>PPO is actually designed to allow this kind of parallelisation as it uses <em>trajectory segments</em> with a fixed size of <span class="math-container">$T$</span> to collect data, e.g. 60 seconds for OpenAI Five, where <span class="math-container">$T$</span> is supposed to be "much less than the episode length" (p.5 of <a href="http://arxiv.org/pdf/1707.06347v2" rel="nofollow noreferrer">PPO paper</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-17T22:18:25.743Z
Answers.Text: So does it update each worker agent after each policy update?
  And is this at all related to 'batched evironments' like can be seen [here](https://www.tensorflow.org/agents/api_docs/python/tf_agents/environments/BatchedPyEnvironment?hl=en)
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-12T13:14:33.383Z
Answers.LastActivityDate: 2020-02-12T13:14:33.383Z
Answers.UserDisplayName: Gregor
Answers.Body: <p>The paper <a href="https://arxiv.org/pdf/1912.06680v1.pdf" rel="nofollow noreferrer">Dota 2 with Large Scale Deep Reinforcement Learning</a> goes into greater detail than the initial blog posts. </p>
  
  <p>They call their distributed training framework <em>Rapid</em>, which is also used in some of their robotics work, such as the paper <a href="https://arxiv.org/pdf/1808.00177.pdf" rel="nofollow noreferrer">Learning Dexterous In-Hand Manipulation</a>, where they discuss a smaller scale deployment of Rapid (as compared to Dota2/OpenAI V) in section 4.3.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-19T19:37:50.087Z
Answers.LastActivityDate: 2020-02-20T20:06:12.830Z
Answers.LastEditDate: 2020-02-20T20:06:12.830Z
Answers.UserDisplayName: crizCraig
Body: <p>There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory buffers (this is how A3C works, for example).</p>
  
  <p>However, there are methods, like PPO, that use batch training on purpose. How is parallelism usually implemented for algorithms that still use batch training?</p>
  
  <p>Are gradients accumulated over parallel workers and the combined? Is there another way? What are the benefits of doing parallelism one way over another?</p>
  
Tags: <reinforcement-learning><actor-critic-methods><implementation><proximal-policy-optimization>
Title: How is parallelism implemented in RL algorithms like PPO?
UserDisplayName: alex vdk
ThreadText: # How is parallelism implemented in RL algorithms like PPO?
  
  (Asked by: username_0 on 2019-04-30)
  
  There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory buffers (this is how A3C works, for example).
  
  However, there are methods, like PPO, that use batch training on purpose. How is parallelism usually implemented for algorithms that still use batch training?
  
  Are gradients accumulated over parallel workers and the combined? Is there another way? What are the benefits of doing parallelism one way over another?
  
  **Tags:** reinforcement-learning, actor-critic-methods, implementation, proximal-policy-optimization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-02-12. Score: 3
  
  OpenAI have a post on that: <https://openai.com/blog/openai-five/>
  
  They use a myriad of *rollout workers* that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.
  
  PPO is actually designed to allow this kind of parallelisation as it uses *trajectory segments* with a fixed size of $T$ to collect data, e.g. 60 seconds for OpenAI Five, where $T$ is supposed to be "much less than the episode length" (p.5 of [PPO paper](http://arxiv.org/pdf/1707.06347v2)).
  
  > **Comments:**
  > **username_2**: So does it update each worker agent after each policy update? And is this at all related to 'batched evironments' like can be seen [here](https://www.tensorflow.org/agents/api_docs/python/tf_agents/environments/BatchedPyEnvironment?hl=en)  
  
  ---
  
  ### Answer by username_3 on 2020-02-19. Score: 1
  
  The paper [Dota 2 with Large Scale Deep Reinforcement Learning](https://arxiv.org/pdf/1912.06680v1.pdf) goes into greater detail than the initial blog posts.
  
  They call their distributed training framework *Rapid*, which is also used in some of their robotics work, such as the paper [Learning Dexterous In-Hand Manipulation](https://arxiv.org/pdf/1808.00177.pdf), where they discuss a smaller scale deployment of Rapid (as compared to Dota2/OpenAI V) in section 4.3.
  
  
  ---
  
  