Answers.Body: <p>Here is the <a href="https://github.com/JeremieGauthier/AI_Exercices/commit/5a54cbcb53887321b272f0a37d149d3d9c3f7304" rel="nofollow noreferrer">commit</a></p>
  
  <p>I fixed few minor errors, but the major one was when I saw what the line <code>histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs)</code> was doing. It was just repeating the same queue.</p>
  
  <pre><code>In [2]: histories = [deque(maxlen=5)] * 4                                       
  
  In [3]: histories                                                               
  Out[3]: [deque([]), deque([]), deque([]), deque([])]
  
  In [4]: histories[0].append(1)                                                  
  
  In [5]: histories                                                               
  Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]
  </code></pre>
  
  <p>So I just replace it by <code>histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]</code>. That fixed my problem.</p>
  
  <pre><code>In [7]: histories = [deque(maxlen=5) for i in range(4)]                         
  
  In [8]: histories                                                               
  Out[8]: [deque([]), deque([]), deque([]), deque([])]
  
  In [9]: histories[0].append(1)                                                  
  
  In [10]: histories                                                              
  Out[10]: [deque([1]), deque([]), deque([]), deque([])]
  </code></pre>
  
  <p>The curve representing the mean reward looks like </p>
  
  <p><a href="https://i.sstatic.net/eDc8z.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eDc8z.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-19T15:41:51.693Z
Answers.LastActivityDate: 2020-05-20T00:43:35.723Z
Answers.LastEditDate: 2020-05-20T00:43:35.723Z
Answers.UserDisplayName: jgauth
Body: <p>I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different.</p>
  
  <ol>
  <li><a href="https://colab.research.google.com/drive/12YQO4r9v7aFSMqE47Vxl_4ku-c4We3B2?usp=sharing" rel="nofollow noreferrer">https://colab.research.google.com/drive/12YQO4r9v7aFSMqE47Vxl_4ku-c4We3B2?usp=sharing</a></li>
  </ol>
  
  <p>The above code is from the following Github repository: <a href="https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py" rel="nofollow noreferrer">https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py</a> It converged perfectly well! </p>
  
  <p>You can find an explanation in Maxim Lapan's book <em>Deep Reinforcement Learning Hands-on</em> page 269</p>
  
  <p>Here is the mean reward curve : </p>
  
  <p><a href="https://i.sstatic.net/pGuDS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pGuDS.png" alt="enter image description here"></a></p>
  
  <ol start="2">
  <li><a href="https://colab.research.google.com/drive/1jkZtk_-kR1Mls9WMbX6l_p1bckph8x1c?usp=sharing" rel="nofollow noreferrer">https://colab.research.google.com/drive/1jkZtk_-kR1Mls9WMbX6l_p1bckph8x1c?usp=sharing</a></li>
  </ol>
  
  <p>The above implementation has been created by me based on the Maxim Lapan's book. However, the code is not converging. There's a small portion of my code that is wrong, but I can't point out what it is. I've been working on that near a week now.</p>
  
  <p>Here is the mean reward curve : </p>
  
  <p><a href="https://i.sstatic.net/VBqYt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VBqYt.png" alt="enter image description here"></a></p>
  
  <p>Can someone tell me the problem portion of the code and how can I fix it?</p>
  
  <p><strong>UPDATE 1</strong></p>
  
  <p>I have decided to test my code with a simpler environment, i.e. <code>Cartpole-v0</code>.</p>
  
  <p>Here is the code : <a href="https://colab.research.google.com/drive/1zL2sy628-J4V1a_NSW2W6MpYinYJSyyZ?usp=sharing" rel="nofollow noreferrer">https://colab.research.google.com/drive/1zL2sy628-J4V1a_NSW2W6MpYinYJSyyZ?usp=sharing</a></p>
  
  <p>Even that code doesn't seem to converge. Still can't see where is my problem. </p>
  
  <p><strong>UPDATE 2</strong></p>
  
  <p>I think the bug might be in the <a href="https://l.facebook.com/l.php?u=https%3A%2F%2Fgithub.com%2FJeremieGauthier%2FAI_Exercices%2Fblob%2Fmaster%2FAtari_Breakout%2FBreakout_A2C%2Fexperience.py%3Ffbclid%3DIwAR1wP3QhgYQ7Bd2rtYcTTzi5Ne0o9cJuxTJAOyTXhYXReDQq9JVAUCR6pyM&amp;h=AT1CW-RwYDgt1g5EmMSgcNUwdjqxy9yDYkUMaBihpuL9dnAEJGkOWfwIQSdLoQYtqlCWo4PFWEDol8oLVu67oD2TWLqbn9EChoE_Sx5-7Aw5OnFj58Z5eKno1mpk7g" rel="nofollow noreferrer">ExperienceSource</a> class or in the <a href="https://l.facebook.com/l.php?u=https%3A%2F%2Fgithub.com%2FJeremieGauthier%2FAI_Exercices%2Fblob%2Fmaster%2FAtari_Breakout%2FBreakout_A2C%2Fagent.py%3Ffbclid%3DIwAR0RsFusMNhK0Qo2H4bE0_Szlu4rVU7imhsh1m3WYbb531pnc_xe2VCpFvY&amp;h=AT1CW-RwYDgt1g5EmMSgcNUwdjqxy9yDYkUMaBihpuL9dnAEJGkOWfwIQSdLoQYtqlCWo4PFWEDol8oLVu67oD2TWLqbn9EChoE_Sx5-7Aw5OnFj58Z5eKno1mpk7g" rel="nofollow noreferrer">Agent</a> class. </p>
  
  <p><strong>UPDATE 3</strong></p>
  
  <p>The following <a href="https://ai.stackexchange.com/questions/21073/once-the-environments-are-vectorized-how-do-i-have-to-gather-immediate-experien?noredirect=1#comment32363_21073">question</a> will help you understand the classes <code>ExperienceSource</code> and <code>ExperienceSourceFirstLast</code>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T19:48:46.497Z
Comments.Text: Note that programming issues are off-topic here. This issue is in the context of RL, so I may not close it.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-15T19:52:20.290Z
Comments.Text: Thanks @nbro! I will keep that in mind. As you allow me, I will leave my question active. Otherwise, I will delete it.
Comments.UserDisplayName: jgauth
Tags: <reinforcement-learning><pytorch><actor-critic-methods><convergence>
Title: Why isn't my implementation of A2C for the the atari pong game converging?
UserDisplayName: jgauth
ThreadText: # Why isn't my implementation of A2C for the the atari pong game converging?
  
  (Asked by: username_0 on 2020-05-15)
  
  I have two different implementations with PyTorch of the Atari Pong game using A2C algorithm. Both implementations are similar, but some portion are different.
  
  1. <https://colab.research.google.com/drive/12YQO4r9v7aFSMqE47Vxl_4ku-c4We3B2?usp=sharing>
  
  The above code is from the following Github repository: <https://github.com/PacktPublishing/Deep-Reinforcement-Learning-Hands-On/blob/master/Chapter10/02_pong_a2c.py> It converged perfectly well!
  
  You can find an explanation in Maxim Lapan's book *Deep Reinforcement Learning Hands-on* page 269
  
  Here is the mean reward curve :
  
  [![enter image description here](https://i.sstatic.net/pGuDS.png)](https://i.sstatic.net/pGuDS.png)
  
  2. <https://colab.research.google.com/drive/1jkZtk_-kR1Mls9WMbX6l_p1bckph8x1c?usp=sharing>
  
  The above implementation has been created by me based on the Maxim Lapan's book. However, the code is not converging. There's a small portion of my code that is wrong, but I can't point out what it is. I've been working on that near a week now.
  
  Here is the mean reward curve :
  
  [![enter image description here](https://i.sstatic.net/VBqYt.png)](https://i.sstatic.net/VBqYt.png)
  
  Can someone tell me the problem portion of the code and how can I fix it?
  
  **UPDATE 1**
  
  I have decided to test my code with a simpler environment, i.e. `Cartpole-v0`.
  
  Here is the code : <https://colab.research.google.com/drive/1zL2sy628-J4V1a_NSW2W6MpYinYJSyyZ?usp=sharing>
  
  Even that code doesn't seem to converge. Still can't see where is my problem.
  
  **UPDATE 2**
  
  I think the bug might be in the [ExperienceSource](https://l.facebook.com/l.php?u=https%3A%2F%2Fgithub.com%2FJeremieGauthier%2FAI_Exercices%2Fblob%2Fmaster%2FAtari_Breakout%2FBreakout_A2C%2Fexperience.py%3Ffbclid%3DIwAR1wP3QhgYQ7Bd2rtYcTTzi5Ne0o9cJuxTJAOyTXhYXReDQq9JVAUCR6pyM&h=AT1CW-RwYDgt1g5EmMSgcNUwdjqxy9yDYkUMaBihpuL9dnAEJGkOWfwIQSdLoQYtqlCWo4PFWEDol8oLVu67oD2TWLqbn9EChoE_Sx5-7Aw5OnFj58Z5eKno1mpk7g) class or in the [Agent](https://l.facebook.com/l.php?u=https%3A%2F%2Fgithub.com%2FJeremieGauthier%2FAI_Exercices%2Fblob%2Fmaster%2FAtari_Breakout%2FBreakout_A2C%2Fagent.py%3Ffbclid%3DIwAR0RsFusMNhK0Qo2H4bE0_Szlu4rVU7imhsh1m3WYbb531pnc_xe2VCpFvY&h=AT1CW-RwYDgt1g5EmMSgcNUwdjqxy9yDYkUMaBihpuL9dnAEJGkOWfwIQSdLoQYtqlCWo4PFWEDol8oLVu67oD2TWLqbn9EChoE_Sx5-7Aw5OnFj58Z5eKno1mpk7g) class.
  
  **UPDATE 3**
  
  The following [question](https://ai.stackexchange.com/questions/21073/once-the-environments-are-vectorized-how-do-i-have-to-gather-immediate-experien?noredirect=1#comment32363_21073) will help you understand the classes `ExperienceSource` and `ExperienceSourceFirstLast`.
  
  **Tags:** reinforcement-learning, pytorch, actor-critic-methods, convergence
  
  > **Comments:**
  > **username_1**: Note that programming issues are off-topic here. This issue is in the context of RL, so I may not close it.  
  > **username_0**: Thanks @username_1! I will keep that in mind. As you allow me, I will leave my question active. Otherwise, I will delete it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-05-19. Score: 2 | Accepted answer
  
  Here is the [commit](https://github.com/JeremieGauthier/AI_Exercices/commit/5a54cbcb53887321b272f0a37d149d3d9c3f7304)
  
  I fixed few minor errors, but the major one was when I saw what the line `histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs)` was doing. It was just repeating the same queue.
  
  ```
  In [2]: histories = [deque(maxlen=5)] * 4                                       
  
  In [3]: histories                                                               
  Out[3]: [deque([]), deque([]), deque([]), deque([])]
  
  In [4]: histories[0].append(1)                                                  
  
  In [5]: histories                                                               
  Out[5]: [deque([1]), deque([1]), deque([1]), deque([1])]
  
  ```
  
  So I just replace it by `histories = [deque(maxlen=self.reward_steps) for i in range(len(self.env.envs))]`. That fixed my problem.
  
  ```
  In [7]: histories = [deque(maxlen=5) for i in range(4)]                         
  
  In [8]: histories                                                               
  Out[8]: [deque([]), deque([]), deque([]), deque([])]
  
  In [9]: histories[0].append(1)                                                  
  
  In [10]: histories                                                              
  Out[10]: [deque([1]), deque([]), deque([]), deque([])]
  
  ```
  
  The curve representing the mean reward looks like
  
  [![enter image description here](https://i.sstatic.net/eDc8z.png)](https://i.sstatic.net/eDc8z.png)
  
  
  ---
  
  