Answers.Body: <blockquote>
    <p>What is the advantage of using more than one environment with a single agent?</p>
  </blockquote>
  
  <p>There are two main advantages to this approach:</p>
  
  <ul>
  <li><p>The dataset for training is closer to the <a href="https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables" rel="nofollow noreferrer">independent, identically distributed (i.i.d.) ideal</a>, important for theoretical and practical reasons when training a neural network. Samples taken from a single trajectory are not independent, but instead are correlated due to rules of the environment - so using a single trajectory is furthest from i.i.d. This is a similar motivation to use of experience replay tables for DQN variants of Q-learning. However, experience replay is inherently off-policy, so not a good fit to A2C or A3C that need samples taken when acting under the current policy.</p></li>
  <li><p>Collecting experience is often a major bottleneck in training RL agents. Being able to do so in parallel in a distributed environment can significantly speed up the training process.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-04T14:14:48.187Z
Answers.LastActivityDate: 2020-05-11T06:50:36.017Z
Answers.LastEditDate: 2020-05-11T06:50:36.017Z
Answers.UserDisplayName: Neil Slater
Body: <pre><code> make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4"))
   envs = [make_env() for _ in range(NUM_ENVS)]
  </code></pre>
  
  <p>Here is a <a href="https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py" rel="nofollow noreferrer">code</a> you can look at.</p>
  
  <p>The two above lines of code create multiple environments for the game of Atari Pong with the A2C algorithm. </p>
  
  <p>I understand why it is very useful to have multiple agents working on different instances of the same environment as it is presented in A3C (i.e. an asynchronous version of A2C). However, in the above code, it has a single agent working on different instances of the same environment.</p>
  
  <p>What is the advantage of using more than one environment with a single agent?</p>
  
  <p><strong>UPDATE</strong></p>
  
  <pre><code>class GymEnvVec:
      def __init__(self, name, n_envs, seed):
          self.envs = [gym.make(name) for i in range(n_envs)]
          [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]
  
      def reset(self):
          return [env.reset() for env in self.envs]
  
      def step(self, actions):
          return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
  </code></pre>
  
Tags: <reinforcement-learning><actor-critic-methods><environment><advantage-actor-critic>
Title: What is the advantage of using more than one environment with the advantage actor-critic?
UserDisplayName: jgauth
ThreadText: # What is the advantage of using more than one environment with the advantage actor-critic?
  
  (Asked by: username_0 on 2020-05-03)
  
  make_env = lambda: ptan.common.wrappers.wrap_dqn(gym.make("PongNoFrameskip-v4"))
   envs = [make_env() for _ in range(NUM_ENVS)]
  
  Here is a code you can look at.
  The two above lines of code create multiple environments for the game of Atari Pong with the A2C algorithm. 
  I understand why it is very useful to have multiple agents working on different instances of the same environment as it is presented in A3C (i.e. an asynchronous version of A2C). However, in the above code, it has a single agent working on different instances of the same environment.
  What is the advantage of using more than one environment with a single agent?
  UPDATE
  class GymEnvVec:
      def __init__(self, name, n_envs, seed):
          self.envs = [gym.make(name) for i in range(n_envs)]
          [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]
  
      def reset(self):
          return [env.reset() for env in self.envs]
  
      def step(self, actions):
          return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
  
  **Tags:** reinforcement-learning, actor-critic-methods, environment, advantage-actor-critic
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-04. Score: 2 | Accepted answer
  
  > What is the advantage of using more than one environment with a single agent?
  
  There are two main advantages to this approach:
  
  * The dataset for training is closer to the [independent, identically distributed (i.i.d.) ideal](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables), important for theoretical and practical reasons when training a neural network. Samples taken from a single trajectory are not independent, but instead are correlated due to rules of the environment - so using a single trajectory is furthest from i.i.d. This is a similar motivation to use of experience replay tables for DQN variants of Q-learning. However, experience replay is inherently off-policy, so not a good fit to A2C or A3C that need samples taken when acting under the current policy.
  * Collecting experience is often a major bottleneck in training RL agents. Being able to do so in parallel in a distributed environment can significantly speed up the training process.
  
  
  ---
  
  