Answers.Body: <p>I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from environment (see more state space). With single agent you can't really achieve that, you would have a single exploration policy, single parameter set for the agent and most of the time you would be seeing similar states (at least after a while). You would be speeding up your learning process but that's just because you're running multiple environments in parallel (compared to the regular actor-critic or Q-learning). I think quality of learning would be better with multiple different actors.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T11:40:40.010Z
Answers.Text: In addition, the multiple agents approach is used for similar reasons to DQN's experience replay - it helps to reduce correlation within training batches. This can be important for stability of a neural network within the A2C framework.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-03T01:38:28.777Z
Answers.Text: Hi, I still feel confused. In general, A2C is implemented with an on-policy policy-gradient algorithm, If we use different parameters for different agents, then the learning is not on-policy at all. Why would we do that? BTW, @NeilSlater, I cannot see how multiple agents reduce correlation in the way multiple environments do not. Could you please shed some light on it?
Answers.UserDisplayName: Maybe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-03T06:43:38.177Z
Answers.Text: @SherwinChen: There is no difference between "multiple agents" and "multiple environments" by default, unless you are deliberately connecting the agents in the former (e.g. forcing them to the same state on each time step) or deliberately changing the environments in the latter (e.g. changing the state transition rules). Neither of those things are suggested here. The point is to reduce correlation. In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-03T07:05:13.420Z
Answers.Text: @NeilSlater, thanks for responding, but I'm sorry that I cannot really understand what you were trying to convey at the end: "In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources." I've updated my question to include a simple environment vector, I cannot see why the data will be less i.i.d. than that collected by multiple agents. I'm looking for your response
Answers.UserDisplayName: Maybe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-03T09:20:13.167Z
Answers.Text: @SherwinChen: A single agent's vectors will be correlated. E.g. you can predict the next observation to train from with some degree of accuracy better than random, even knowing nothing about the agent. Whilst if you take the next observation from a different agent, your ability to predict what the state is in advance is reduced. Self-correlated data is not i.i.d. and you want to avoid that, especially when training incrementally using online learners such as neural networks
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-03T23:57:48.323Z
Answers.Text: Hi @NeilSlater, and thanks again. I'm still quite confused about why, in the case of using environment vector, we can predict the next observation to train from in the way that we cannot do the same to multiple agents. In A2C, even we use multiple agents, they are just nothing but copies of the learning agent since every time we update the learning agent, we synchronize the updated parameters with all agents. Environments are different in both cases since we use different seeds. What on earth makes the difference? Or am I misunderstanding something?
Answers.UserDisplayName: Maybe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-04T06:45:46.937Z
Answers.Text: @SherwinChen: Clearly I am not able to explain this in your terms in the comments. Consider asking another question on the site
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T02:36:20.740Z
Answers.Text: @NeilSlater Awesome explanation Neil!
Answers.UserDisplayName: jgauth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T08:09:27.980Z
Answers.LastActivityDate: 2019-03-31T08:09:27.980Z
Answers.UserDisplayName: Brale
Answers.Body: <p>When you run multiple actors you actually make copies of your current agent and run these copies in non-vector environments. The environments are usually also the same.</p>
  <p>In the case of Atari, yes of course, using a vectorized environment is much more efficient. But if you want to train an agent to play DotA, or drive a car, then there is no vectorized environment.</p>
  <p>Running <em>different</em> actors instead of copies of the same agent is not really possible with on-policy algorithms. You have to collect data using one and the same policy, otherwise it is not on-policy.</p>
  <p>A slight exception to this is the case of <a href="https://arxiv.org/abs/1602.01783" rel="nofollow noreferrer">asynchronous training</a>. In this case you start with the same copies of your agent, but update these copies asynchronously, and only once in a while sync the policies. However, you still have to make sure the actors' policies don't deviate too much from one another. But note that the synchronous approach maybe performs better than the asynchronous. And again, in case you can vectorize your environment I don't really see the need to do this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-25T22:05:35.953Z
Answers.LastActivityDate: 2023-08-26T11:25:37.660Z
Answers.LastEditDate: 2023-08-26T11:25:37.660Z
Answers.UserDisplayName: pi-tau
Body: <p>I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector? </p>
  
  <p>I personally think this will be more efficient since now all actions can be calculated together by only going through the network once. I've done some tests, and this seems to work fine in my test. One reason I can think of to use multiple actors is implementing the algorithm across many machines, in which case we can have one agent on a machine. What else reason should we prefer multiple actors?</p>
  
  <p>As an example of environment vector based on OpenAI's gym</p>
  
  
  
  <pre><code>class GymEnvVec:
      def __init__(self, name, n_envs, seed):
          self.envs = [gym.make(name) for i in range(n_envs)]
          [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]
  
      def reset(self):
          return [env.reset() for env in self.envs]
  
      def step(self, actions):
          return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-31T04:35:28.363Z
Comments.Text: Can you clarify what you mean by an "environment vector"?
Comments.UserDisplayName: Philip Raeisghasem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-31T09:13:28.170Z
Comments.Text: Hi @PhilipRaeisghasem, please refer to [this](https://github.com/ray-project/ray/blob/master/python/ray/rllib/env/vector_env.py) for an example.
Comments.UserDisplayName: Maybe
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T13:46:46.953Z
Comments.Text: @Maybe Where did you take the code related to `GymEnvVec`?
Comments.UserDisplayName: jgauth
Tags: <reinforcement-learning><actor-critic-methods><gym><advantage-actor-critic>
Title: What is the difference between A2C and running an agent in an environment vector?
UserDisplayName: Maybe
ThreadText: # What is the difference between A2C and running an agent in an environment vector?
  
  (Asked by: username_0 on 2019-03-31)
  
  I've implemented A2C. I'm now wondering why would we have multiple actors walk around the environment and gather rewards, why not just have a single agent run in an environment vector?
  
  I personally think this will be more efficient since now all actions can be calculated together by only going through the network once. I've done some tests, and this seems to work fine in my test. One reason I can think of to use multiple actors is implementing the algorithm across many machines, in which case we can have one agent on a machine. What else reason should we prefer multiple actors?
  
  As an example of environment vector based on OpenAI's gym
  
  ```
  class GymEnvVec:
      def __init__(self, name, n_envs, seed):
          self.envs = [gym.make(name) for i in range(n_envs)]
          [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]
  
      def reset(self):
          return [env.reset() for env in self.envs]
  
      def step(self, actions):
          return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
  
  ```
  
  **Tags:** reinforcement-learning, actor-critic-methods, gym, advantage-actor-critic
  
  > **Comments:**
  > **username_1**: Can you clarify what you mean by an "environment vector"?  
  > **username_0**: Hi @username_1, please refer to [this](https://github.com/ray-project/ray/blob/master/python/ray/rllib/env/vector_env.py) for an example.  
  > **username_2**: @username_0 Where did you take the code related to `GymEnvVec`?  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-03-31. Score: 2
  
  I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from environment (see more state space). With single agent you can't really achieve that, you would have a single exploration policy, single parameter set for the agent and most of the time you would be seeing similar states (at least after a while). You would be speeding up your learning process but that's just because you're running multiple environments in parallel (compared to the regular actor-critic or Q-learning). I think quality of learning would be better with multiple different actors.
  
  > **Comments:**
  > **username_4**: In addition, the multiple agents approach is used for similar reasons to DQN's experience replay - it helps to reduce correlation within training batches. This can be important for stability of a neural network within the A2C framework.  
  > **username_0**: Hi, I still feel confused. In general, A2C is implemented with an on-policy policy-gradient algorithm, If we use different parameters for different agents, then the learning is not on-policy at all. Why would we do that? BTW, @username_4, I cannot see how multiple agents reduce correlation in the way multiple environments do not. Could you please shed some light on it?  
  > **username_4**: @SherwinChen: There is no difference between "multiple agents" and "multiple environments" by default, unless you are deliberately connecting the agents in the former (e.g. forcing them to the same state on each time step) or deliberately changing the environments in the latter (e.g. changing the state transition rules). Neither of those things are suggested here. The point is to reduce correlation. In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources.  
  > **username_0**: @username_4, thanks for responding, but I'm sorry that I cannot really understand what you were trying to convey at the end: "In your question you suggest collecting and using data for the variations sequentially - a training batch of your data will be less i.i.d. than a batch which collects from multiple sources." I've updated my question to include a simple environment vector, I cannot see why the data will be less i.i.d. than that collected by multiple agents. I'm looking for your response  
  > **username_4**: @SherwinChen: A single agent's vectors will be correlated. E.g. you can predict the next observation to train from with some degree of accuracy better than random, even knowing nothing about the agent. Whilst if you take the next observation from a different agent, your ability to predict what the state is in advance is reduced. Self-correlated data is not i.i.d. and you want to avoid that, especially when training incrementally using online learners such as neural networks  
  > **username_0**: Hi @username_4, and thanks again. I'm still quite confused about why, in the case of using environment vector, we can predict the next observation to train from in the way that we cannot do the same to multiple agents. In A2C, even we use multiple agents, they are just nothing but copies of the learning agent since every time we update the learning agent, we synchronize the updated parameters with all agents. Environments are different in both cases since we use different seeds. What on earth makes the difference? Or am I misunderstanding something?  
  > **username_4**: @SherwinChen: Clearly I am not able to explain this in your terms in the comments. Consider asking another question on the site  
  > **username_2**: @username_4 Awesome explanation username_4!  
  
  ---
  
  ### Answer by username_5 on 2023-08-25. Score: 0
  
  When you run multiple actors you actually make copies of your current agent and run these copies in non-vector environments. The environments are usually also the same.
  
  In the case of Atari, yes of course, using a vectorized environment is much more efficient. But if you want to train an agent to play DotA, or drive a car, then there is no vectorized environment.
  
  Running *different* actors instead of copies of the same agent is not really possible with on-policy algorithms. You have to collect data using one and the same policy, otherwise it is not on-policy.
  
  A slight exception to this is the case of [asynchronous training](https://arxiv.org/abs/1602.01783). In this case you start with the same copies of your agent, but update these copies asynchronously, and only once in a while sync the policies. However, you still have to make sure the actors' policies don't deviate too much from one another. But note that the synchronous approach username_0 performs better than the asynchronous. And again, in case you can vectorize your environment I don't really see the need to do this.
  
  
  ---
  
  