Answers.Body: <pre><code>class ExperienceSource():
      def __init__(self, env, agent, reward_steps):
          self.env = env
          self.agent = agent
          self.reward_steps = reward_steps
  
      def __iter__(self):
          histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
          states = self.env.reset()
  
          while True:
  
              for idx, env in enumerate(self.env.envs):
                  action = self.agent.choose_action(states[idx])
                  state, reward, done, _ = env.step(action)
  
                  current_rewards[idx] += reward
                  histories[idx].append(Experience(state, action, reward, done))
  
                  if len(histories[idx]) == self.reward_steps:
                      yield tuple(histories[idx])
  
                  if done: 
                      yield tuple(histories[idx])
                      state = env.reset()
  </code></pre>
  
  <p>Be aware that <code>self.reward_steps</code> is simply the value defined by <code>N-1</code> in the following formula <span class="math-container">$$Q(s,a) = \sum_{i=0}^{N-1} \gamma^i r_i
   + \gamma^N V(s_N)$$</span> and <code>self.env</code> is simply an instance of <code>GymEnvVec</code> class from the question.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-12T14:43:40.343Z
Answers.LastActivityDate: 2020-05-18T09:58:31.163Z
Answers.LastEditDate: 2020-05-18T09:58:31.163Z
Answers.UserDisplayName: jgauth
Body: <p>My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the correlation between samples (i.i.d), I need to have an agent interacting with several environments. </p>
  
  <pre><code>class GymEnvVec():
  
      def __init__(self, env_name, n_envs, seed=0):
          make_env = lambda: gym.make(env_name)
          self.envs = [make_env() for _ in range(n_envs)]
          [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]
  
      def reset(self):
          return [env.reset() for env in self.envs]
  
      def step(self, actions):
          return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
  </code></pre>
  
  <p>I can use the class <code>GymEnvVec</code> to vectorize my environment. </p>
  
  <p>So I can set my environments with </p>
  
  <pre><code>envs = GymEnvVec(env_name="Breakout-v0", n_envs=50)
  </code></pre>
  
  <p>I can get my first observations with</p>
  
  <pre><code>observations = envs.reset()
  </code></pre>
  
  <p>Pick some actions with </p>
  
  <pre><code>actions = agent.choose_actions(observations)
  </code></pre>
  
  <p>The <code>choose_actions</code> method might look like </p>
  
  <pre><code>def choose_actions(self, states):
          assert isinstance(states, (list, tuple))
  
          actions = []
          for state in states:
              probabilities  = F.softmax(self.network(state)[0])
              action_probs = T.distributions.Categorical(probabilities)
              actions.append(action_probs.sample())
  
          return [action.item() for action in actions] 
  </code></pre>
  
  <p>Finally, the environments will spit the next_states, rewards and if it is done with</p>
  
  <pre><code>next_states, rewards, dones, _ = env.step(actions)
  </code></pre>
  
  <p>It is at this point I am a bit confused. I think I need to gather immediate experiences, batch altogether and forward it to the agent. My problem is probably with the "gather immediate experiences". </p>
  
  <p>I propose a solution, but I am far from being sure it is a good answer. At each iteration, I think I must take a random number with </p>
  
  <pre><code>nb = random.randint(0, len(n_envs)-1)
  </code></pre>
  
  <p>and put the experience in history with </p>
  
  <pre><code>history.append(Experience(state=states[nb], actions[nb], rewards[nb], dones[nb]))
  </code></pre>
  
  <p>Am I wrong? Can you tell me what I should do?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-11T17:08:03.830Z
Comments.Text: I think you'd need multiple histories, one for each env
Comments.UserDisplayName: Santiago Benoit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-12T14:44:45.170Z
Comments.Text: @SantiagoBenoit I answered my own question. Can you tell me if it is a fine way to do it?
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-18T05:17:05.457Z
Comments.Text: Looks like the right idea - btw I made an edit to your histories list because it was just repeating the same queue
Comments.UserDisplayName: Santiago Benoit
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-18T10:11:40.347Z
Comments.Text: I have seen what you meant. indeed, if I run `histories = [deque(maxlen=5)] * 4` and  `histories[0].append(1)`, the output is `[deque([1]), deque([1]), deque([1]), deque([1])]` which is very annoying. Thanks @SantiagoBenoit! Can you explain that behavior? As you can see, instead I wanted `[deque([1]), deque([]), deque([]), deque([])]`
Comments.UserDisplayName: jgauth
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-19T04:50:43.953Z
Comments.Text: In Python, multiplication operator on list doesn't create deep copies of objects. Basically what is happening is instead of creating new queues, it is creating references to the same queue.
Comments.UserDisplayName: Santiago Benoit
Tags: <reinforcement-learning><python><pytorch><actor-critic-methods><environment>
Title: Once the environments are vectorized, how do I have to gather immediate experiences for the agent?
UserDisplayName: jgauth
ThreadText: # Once the environments are vectorized, how do I have to gather immediate experiences for the agent?
  
  (Asked by: username_0 on 2020-05-11)
  
  My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the correlation between samples (i.i.d), I need to have an agent interacting with several environments. 
  class GymEnvVec():
  
      def __init__(self, env_name, n_envs, seed=0):
          make_env = lambda: gym.make(env_name)
          self.envs = [make_env() for _ in range(n_envs)]
          [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]
  
      def reset(self):
          return [env.reset() for env in self.envs]
  
      def step(self, actions):
          return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
  
  I can use the class GymEnvVec to vectorize my environment. 
  So I can set my environments with 
  envs = GymEnvVec(env_name="Breakout-v0", n_envs=50)
  
  I can get my first observations with
  observations = envs.reset()
  
  Pick some actions with 
  actions = agent.choose_actions(observations)
  
  The choose_actions method might look like 
  def choose_actions(self, states):
          assert isinstance(states, (list, tuple))
  
          actions = []
          for state in states:
              probabilities  = F.softmax(self.network(state)[0])
              action_probs = T.distributions.Categorical(probabilities)
              actions.append(action_probs.sample())
  
          return [action.item() for action in actions] 
  
  Finally, the environments will spit the next_states, rewards and if it is done with
  next_states, rewards, dones, _ = env.step(actions)
  
  It is at this point I am a bit confused. I think I need to gather immediate experiences, batch altogether and forward it to the agent. My problem is probably with the "gather immediate experiences". 
  I propose a solution, but I am far from being sure it is a good answer. At each iteration, I think I must take a random number with 
  nb = random.randint(0, len(n_envs)-1)
  
  and put the experience in history with 
  history.append(Experience(state=states[nb], actions[nb], rewards[nb], dones[nb]))
  
  Am I wrong? Can you tell me what I should do?
  
  **Tags:** reinforcement-learning, python, pytorch, actor-critic-methods, environment
  > **Comments:**
  > **username_1**: I think you'd need multiple histories, one for each env  
  > **username_0**: @SantiagoBenoit I answered my own question. Can you tell me if it is a fine way to do it?  
  > **username_1**: Looks like the right idea - btw I made an edit to your histories list because it was just repeating the same queue  
  > **username_0**: I have seen what you meant. indeed, if I run `histories = [deque(maxlen=5)] * 4` and  `histories[0].append(1)`, the output is `[deque([1]), deque([1]), deque([1]), deque([1])]` which is very annoying. Thanks @SantiagoBenoit! Can you explain that behavior? As you can see, instead I wanted `[deque([1]), deque([]), deque([]), deque([])]`  
  > **username_1**: In Python, multiplication operator on list doesn't create deep copies of objects. Basically what is happening is instead of creating new queues, it is creating references to the same queue.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-12. Score: 0 | Accepted answer
  
  class ExperienceSource():
      def __init__(self, env, agent, reward_steps):
          self.env = env
          self.agent = agent
          self.reward_steps = reward_steps
  
      def __iter__(self):
          histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
          states = self.env.reset()
  
          while True:
  
              for idx, env in enumerate(self.env.envs):
                  action = self.agent.choose_action(states[idx])
                  state, reward, done, _ = env.step(action)
  
                  current_rewards[idx] += reward
                  histories[idx].append(Experience(state, action, reward, done))
  
                  if len(histories[idx]) == self.reward_steps:
                      yield tuple(histories[idx])
  
                  if done: 
                      yield tuple(histories[idx])
                      state = env.reset()
  
  Be aware that self.reward_steps is simply the value defined by N-1 in the following formula $$Q(s,a) = \sum_{i=0}^{N-1} \gamma^i r_i
   + \gamma^N V(s_N)$$ and self.env is simply an instance of GymEnvVec class from the question.
  
  
  ---
  
  