Answers.Body: <p>From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to unroll a single episode), yes it can.</p>
  <p>What I do is that I have <span class="math-container">$n$</span> parallel environments unrolling simultaneously and collecting transitions, and I perform <span class="math-container">$n$</span> updates after each parallel step (i.e. all <span class="math-container">$n$</span> parallel envs have collected one transition), in order to keep a 1/1 ratio between stepping and training. I believe it is okay from a theoretical point of view, although I'm not 100% certain. From a practical point of view, it has worked well so far.</p>
  <p>The difference between offline and online collection is that the sampling/training procedures are very different, as in online methods you usually sample a large buffer of samples before performing several epochs of training. Yet, I am curious about what would happen if one was to perform the same for, say, TD3: sample a large buffer of <span class="math-container">$m$</span> transitions, store them in the replay buffer, and then perform <span class="math-container">$m$</span> updates to keep the 1/1 ratio.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-06T02:55:26.503Z
Answers.Text: Have you ever tried to adjust the batch size by multiplying the original batch size for one environment by the number of parallel environments instead in order to maintain 1/1 ratio between stepping and training? I’ve seen one such GitHub implementation of parallelization of off policy algorithms. Obviously this may lead to relatively huge batch sizes; however Open AI is using batch size of tens of millions as reported here https://openai.com/blog/science-of-ai/. But I believe they are applying this in on policy algorithms (since DOTA 2 bots were trained with PPO).
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-13T07:39:51.787Z
Answers.Text: @Mika thanks for accepting the answer. I'm not certain to understand the follow-up question.
  In sequential, I perform one gradient step at every time-step by sampling one mini-batch of size $b$ from my replay buffer.
  In parallel, I still use a mini-batch of size $b$, but I collect $n$ samples in parallel, so I perform $n$ gradient steps at every time-step by sampling $n$ different mini-batches from my replay buffer.
  What you suggest is to perform $n$ gradient steps, but using mini-batches of size $nb$ ?
Answers.UserDisplayName: Scrimbibete
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-30T06:22:00.897Z
Answers.LastActivityDate: 2022-06-30T06:22:00.897Z
Answers.UserDisplayName: Scrimbibete
Body: <p>On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-policy algorithms, like SAC and TD3.</p>
  <p>Is it because the replay memory is kind of a substitute for the parallel data sampling in the on-policy algorithms? Can off-policy algorithms benefit from the parallelization?</p>
  <p>Ray RlLib says <a href="https://docs.ray.io/en/releases-0.7.2/rllib-algorithms.html" rel="nofollow noreferrer">the following</a> for SAC and TD3 regarding the number of workers for collecting samples</p>
  <blockquote>
  <p>This only makes sense to increase if your environment is particularly slow to sample</p>
  </blockquote>
  
Tags: <reinforcement-learning><proximal-policy-optimization><off-policy-methods><on-policy-methods><soft-actor-critic>
Title: Can off-policy algorithms benefit from the parallelization?
UserDisplayName: Mika
ThreadText: # Can off-policy algorithms benefit from the parallelization?
  
  (Asked by: username_0 on 2022-03-27)
  
  On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-policy algorithms, like SAC and TD3.
  
  Is it because the replay memory is kind of a substitute for the parallel data sampling in the on-policy algorithms? Can off-policy algorithms benefit from the parallelization?
  
  Ray RlLib says [the following](https://docs.ray.io/en/releases-0.7.2/rllib-algorithms.html) for SAC and TD3 regarding the number of workers for collecting samples
  
  > This only makes sense to increase if your environment is particularly slow to sample
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, off-policy-methods, on-policy-methods, soft-actor-critic
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-06-30. Score: 2 | Accepted answer
  
  From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to unroll a single episode), yes it can.
  
  What I do is that I have $n$ parallel environments unrolling simultaneously and collecting transitions, and I perform $n$ updates after each parallel step (i.e. all $n$ parallel envs have collected one transition), in order to keep a 1/1 ratio between stepping and training. I believe it is okay from a theoretical point of view, although I'm not 100% certain. From a practical point of view, it has worked well so far.
  
  The difference between offline and online collection is that the sampling/training procedures are very different, as in online methods you usually sample a large buffer of samples before performing several epochs of training. Yet, I am curious about what would happen if one was to perform the same for, say, TD3: sample a large buffer of $m$ transitions, store them in the replay buffer, and then perform $m$ updates to keep the 1/1 ratio.
  
  > **Comments:**
  > **username_0**: Have you ever tried to adjust the batch size by multiplying the original batch size for one environment by the number of parallel environments instead in order to maintain 1/1 ratio between stepping and training? I’ve seen one such GitHub implementation of parallelization of off policy algorithms. Obviously this may lead to relatively huge batch sizes; however Open AI is using batch size of tens of millions as reported here https://openai.com/blog/science-of-ai/. But I believe they are applying this in on policy algorithms (since DOTA 2 bots were trained with PPO).  
  > **username_1**: @username_0 thanks for accepting the answer. I'm not certain to understand the follow-up question. In sequential, I perform one gradient step at every time-step by sampling one mini-batch of size $b$ from my replay buffer. In parallel, I still use a mini-batch of size $b$, but I collect $n$ samples in parallel, so I perform $n$ gradient steps at every time-step by sampling $n$ different mini-batches from my replay buffer. What you suggest is to perform $n$ gradient steps, but using mini-batches of size $nb$ ?  
  
  ---
  
  