<pre><code>class ExperienceSource():
    def __init__(self, env, agent, reward_steps):
        self.env = env
        self.agent = agent
        self.reward_steps = reward_steps

    def __iter__(self):
        histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
        states = self.env.reset()

        while True:

            for idx, env in enumerate(self.env.envs):
                action = self.agent.choose_action(states[idx])
                state, reward, done, _ = env.step(action)

                current_rewards[idx] += reward
                histories[idx].append(Experience(state, action, reward, done))

                if len(histories[idx]) == self.reward_steps:
                    yield tuple(histories[idx])

                if done: 
                    yield tuple(histories[idx])
                    state = env.reset()
</code></pre>

<p>Be aware that <code>self.reward_steps</code> is simply the value defined by <code>N-1</code> in the following formula <span class="math-container">$$Q(s,a) = \sum_{i=0}^{N-1} \gamma^i r_i
 + \gamma^N V(s_N)$$</span> and <code>self.env</code> is simply an instance of <code>GymEnvVec</code> class from the question.</p>

CC BY-SA 4.0
2020-05-12T14:43:40.343Z
2020-05-18T09:58:31.163Z
2020-05-18T09:58:31.163Z
jgauth
<p>My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the correlation between samples (i.i.d), I need to have an agent interacting with several environments. </p>

<pre><code>class GymEnvVec():

    def __init__(self, env_name, n_envs, seed=0):
        make_env = lambda: gym.make(env_name)
        self.envs = [make_env() for _ in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))
</code></pre>

<p>I can use the class <code>GymEnvVec</code> to vectorize my environment. </p>

<p>So I can set my environments with </p>

<pre><code>envs = GymEnvVec(env_name="Breakout-v0", n_envs=50)
</code></pre>

<p>I can get my first observations with</p>

<pre><code>observations = envs.reset()
</code></pre>

<p>Pick some actions with </p>

<pre><code>actions = agent.choose_actions(observations)
</code></pre>

<p>The <code>choose_actions</code> method might look like </p>

<pre><code>def choose_actions(self, states):
        assert isinstance(states, (list, tuple))

        actions = []
        for state in states:
            probabilities  = F.softmax(self.network(state)[0])
            action_probs = T.distributions.Categorical(probabilities)
            actions.append(action_probs.sample())

        return [action.item() for action in actions] 
</code></pre>

<p>Finally, the environments will spit the next_states, rewards and if it is done with</p>

<pre><code>next_states, rewards, dones, _ = env.step(actions)
</code></pre>

<p>It is at this point I am a bit confused. I think I need to gather immediate experiences, batch altogether and forward it to the agent. My problem is probably with the "gather immediate experiences". </p>

<p>I propose a solution, but I am far from being sure it is a good answer. At each iteration, I think I must take a random number with </p>

<pre><code>nb = random.randint(0, len(n_envs)-1)
</code></pre>

<p>and put the experience in history with </p>

<pre><code>history.append(Experience(state=states[nb], actions[nb], rewards[nb], dones[nb]))
</code></pre>

<p>Am I wrong? Can you tell me what I should do?</p>

CC BY-SA 4.0
2020-05-11T17:08:03.830Z
I think you'd need multiple histories, one for each env
Santiago Benoit
CC BY-SA 4.0
2020-05-12T14:44:45.170Z
@SantiagoBenoit I answered my own question. Can you tell me if it is a fine way to do it?
jgauth
CC BY-SA 4.0
2020-05-18T05:17:05.457Z
Looks like the right idea - btw I made an edit to your histories list because it was just repeating the same queue
Santiago Benoit
CC BY-SA 4.0
2020-05-18T10:11:40.347Z
I have seen what you meant. indeed, if I run `histories = [deque(maxlen=5)] * 4` and  `histories[0].append(1)`, the output is `[deque([1]), deque([1]), deque([1]), deque([1])]` which is very annoying. Thanks @SantiagoBenoit! Can you explain that behavior? As you can see, instead I wanted `[deque([1]), deque([]), deque([]), deque([])]`
jgauth
CC BY-SA 4.0
2020-05-19T04:50:43.953Z
In Python, multiplication operator on list doesn't create deep copies of objects. Basically what is happening is instead of creating new queues, it is creating references to the same queue.
Santiago Benoit
<reinforcement-learning><python><pytorch><actor-critic-methods><environment>
Once the environments are vectorized, how do I have to gather immediate experiences for the agent?
jgauth
# Once the environments are vectorized, how do I have to gather immediate experiences for the agent?

(Asked by: username_0 on 2020-05-11)

My main purpose right now is to train an agent using the A2C algorithm to solve the Atari Breakout game. So far I have succeeded to create that code with a single agent and environment. To break the correlation between samples (i.i.d), I need to have an agent interacting with several environments. 
class GymEnvVec():

    def __init__(self, env_name, n_envs, seed=0):
        make_env = lambda: gym.make(env_name)
        self.envs = [make_env() for _ in range(n_envs)]
        [env.seed(seed + 10 * i) for i, env in enumerate(self.envs)]

    def reset(self):
        return [env.reset() for env in self.envs]

    def step(self, actions):
        return list(zip(*[env.step(a) for env, a in zip(self.envs, actions)]))

I can use the class GymEnvVec to vectorize my environment. 
So I can set my environments with 
envs = GymEnvVec(env_name="Breakout-v0", n_envs=50)

I can get my first observations with
observations = envs.reset()

Pick some actions with 
actions = agent.choose_actions(observations)

The choose_actions method might look like 
def choose_actions(self, states):
        assert isinstance(states, (list, tuple))

        actions = []
        for state in states:
            probabilities  = F.softmax(self.network(state)[0])
            action_probs = T.distributions.Categorical(probabilities)
            actions.append(action_probs.sample())

        return [action.item() for action in actions] 

Finally, the environments will spit the next_states, rewards and if it is done with
next_states, rewards, dones, _ = env.step(actions)

It is at this point I am a bit confused. I think I need to gather immediate experiences, batch altogether and forward it to the agent. My problem is probably with the "gather immediate experiences". 
I propose a solution, but I am far from being sure it is a good answer. At each iteration, I think I must take a random number with 
nb = random.randint(0, len(n_envs)-1)

and put the experience in history with 
history.append(Experience(state=states[nb], actions[nb], rewards[nb], dones[nb]))

Am I wrong? Can you tell me what I should do?

**Tags:** reinforcement-learning, python, pytorch, actor-critic-methods, environment

> **Comments:**
> **username_1**: I think you'd need multiple histories, one for each env  
> **username_0**: @SantiagoBenoit I answered my own question. Can you tell me if it is a fine way to do it?  
> **username_1**: Looks like the right idea - btw I made an edit to your histories list because it was just repeating the same queue  
> **username_0**: I have seen what you meant. indeed, if I run `histories = [deque(maxlen=5)] * 4` and  `histories[0].append(1)`, the output is `[deque([1]), deque([1]), deque([1]), deque([1])]` which is very annoying. Thanks @SantiagoBenoit! Can you explain that behavior? As you can see, instead I wanted `[deque([1]), deque([]), deque([]), deque([])]`  
> **username_1**: In Python, multiplication operator on list doesn't create deep copies of objects. Basically what is happening is instead of creating new queues, it is creating references to the same queue.  

---

## 1 answers

### Answer by username_0 on 2020-05-12. Score: 0 | Accepted answer

class ExperienceSource():
    def __init__(self, env, agent, reward_steps):
        self.env = env
        self.agent = agent
        self.reward_steps = reward_steps

    def __iter__(self):
        histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]
        states = self.env.reset()

        while True:

            for idx, env in enumerate(self.env.envs):
                action = self.agent.choose_action(states[idx])
                state, reward, done, _ = env.step(action)

                current_rewards[idx] += reward
                histories[idx].append(Experience(state, action, reward, done))

                if len(histories[idx]) == self.reward_steps:
                    yield tuple(histories[idx])

                if done: 
                    yield tuple(histories[idx])
                    state = env.reset()

Be aware that self.reward_steps is simply the value defined by N-1 in the following formula $$Q(s,a) = \sum_{i=0}^{N-1} \gamma^i r_i
 + \gamma^N V(s_N)$$ and self.env is simply an instance of GymEnvVec class from the question.


---

