Answers.Body: <p>Incase you still haven't been able to resolve the problem, here's a link to the <a href="https://ai.stackexchange.com/a/10481/21513">answer</a> to my own question, which has the step-wise changes I made to achieve a +18 average score saturation using just a 10000 replay buffer size and a normal Double DQN (DDQN), trained for about 700-800 episodes. The updated code can also be found <a href="https://github.com/hridayns/Reinforcement-Learning/tree/master/Atari" rel="nofollow noreferrer">here</a>.</p>
  
  <p>No fancy changes like Prioritized Replay Buffer or any secret hyperparameter changes are required. It's usually something simple, like a small problem with the input preprocessing step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-10T04:04:36.467Z
Answers.LastActivityDate: 2019-02-10T04:04:36.467Z
Answers.UserDisplayName: hridayns
Body: <p>I am in the process of implementing the DQN model from scratch in PyTorch with the target environment of Atari Pong. After a while of tweaking hyper-parameters, I cannot seem to get the model to achieve the performance that is reported in most publications (~ +21 reward; meaning that the agent wins almost every volley). </p>
  
  <p>My most recent results are shown in the following figure. Note that the x axis is episodes (full games to 21), but the total training iterations is ~6.7 million.</p>
  
  <p><a href="https://i.sstatic.net/3TAEJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3TAEJ.png" alt="enter image description here"></a></p>
  
  <p>The specifics of my setup are as follows:</p>
  
  <h3>Model</h3>
  
  <pre><code>class DQN(nn.Module):
      def __init__(self, in_channels, outputs):
          super(DQN, self).__init__()
          self.conv1 = nn.Conv2d(in_channels=in_channels, out_channels=32, kernel_size=8, stride=4)
          self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2)
          self.conv3 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1)
          self.fc1 = nn.Linear(in_features=64*7*7 , out_features=512)
          self.fc2 = nn.Linear(in_features=512, out_features=outputs)
  
      def forward(self, x):
          x = F.relu(self.conv1(x))
          x = F.relu(self.conv2(x))
          x = F.relu(self.conv3(x))
          x = x.view(-1, 64 * 7 * 7)
          x = F.relu(self.fc1(x))
          x = self.fc2(x)
          return x    # return Q values of each action
  </code></pre>
  
  <h3>Hyperparameters</h3>
  
  <ul>
  <li>batch size: 32</li>
  <li>replay memory size: 100000</li>
  <li>initial epsilon: 1.0</li>
  <li>epsilon anneals linearly to 0.02 over 100000 steps</li>
  <li>random warmstart episodes: ~50000</li>
  <li>update target model every: 1000 steps</li>
  <li>optimizer = optim.RMSprop(policy_net.parameters(),  lr=0.0025, alpha=0.9, eps=1e-02, momentum=0.0)</li>
  </ul>
  
  <h3>Additional info</h3>
  
  <ul>
  <li>OpenAI gym Pong-v0 environment</li>
  <li>Feeding model stacks of 4 last observed frames, scaled and cropped to 84x84 such that only the "playing area" is visible.</li>
  <li>Treat losing a volley (end-of-life) as a terminal state in the replay buffer.</li>
  <li>Using <a href="https://pytorch.org/docs/stable/nn.html#smooth-l1-loss" rel="nofollow noreferrer">smooth_l1_loss</a>, which acts as Huber loss</li>
  <li>Clipping gradients between -1 and 1 before optimizing</li>
  <li>I offset the beginning of each episode with 4-30 no-op steps as the papers suggest</li>
  </ul>
  
  <p>Has anyone had a similar experience of getting stuck around 6 - 9 average reward per episode like this? </p>
  
  <p>Any suggestions for changes to hyperparameters or algorithmic nuances would be greatly appreciated!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-26T12:48:33.153Z
Comments.Text: Hi @Mink , I am working on the same project right now, but my average score is capped at -10. Some differences I can see between our implementations are:
  
  * "PongDeterministic-v4" environment which takes care of frame skipping.
  * did not crop the score part on the top, but my frames are 84 x 84.
  * do not treat losing a volley as a terminal state.
  * replay buffer size is 50000. RAM fills up and overflows if I use any more than that. No idea how you managed to store 100000 experience tuples.
  * using l2 loss, was planning to switch to huber loss though.
  
  Do you have a Github repo?
Comments.UserDisplayName: hridayns
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-28T01:50:08.577Z
Comments.Text: @hridayns, to prevent memory overflow, I save the images as uint8 type in my replay buffer and cast (and divide by 255) right before I forward pass it to my model.
Comments.UserDisplayName: Mink
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-28T01:52:26.203Z
Comments.Text: By the way, the GitHub repo is https://github.com/MatthewInkawhich/learnRL
Comments.UserDisplayName: Mink
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-28T11:42:56.767Z
Comments.Text: I have done the same uint8 conversion. I was planning to normalize them by 255 too, but I believe they take up the same memory, whether divided or not. I was thinking I might lose information if I did that. What do you think? Also, I have followed you on github. In regards to your issue, I was thinking maybe it has something to do with your target network update frequency. It could be too low, causing instability during learning? Another thing you could try is a Prioritized Replay buffer.
Comments.UserDisplayName: hridayns
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-28T14:08:31.190Z
Comments.Text: @hridayns thanks for the suggestions, I will try that. Also, I would recommend always normalizing your image inputs between 0 and 1, as it tends to play nicer with the initial weights and activation function.
Comments.UserDisplayName: Mink
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-29T10:23:06.063Z
Comments.Text: Thanks. I will probably implement that then. Along with better initialization schemes for the networks, gradient and reward clipping. You can check out my repo here if you like: https://github.com/hridayns/Reinforcement-Learning/
Comments.UserDisplayName: hridayns
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-29T14:21:36.213Z
Comments.Text: According to this answer [here](https://stackoverflow.com/a/38363141/3623131), using a squared loss function with gradient clipping is equivalent to using a Huber loss function without the gradient clipping. Gradient clipping step may not be required for you since you are using something close to Huber loss already.
Comments.UserDisplayName: hridayns
Tags: <reinforcement-learning><deep-rl><dqn><implementation><atari-games>
Title: DQN stuck at suboptimal policy in Atari Pong task
UserDisplayName: Mink
ThreadText: # DQN stuck at suboptimal policy in Atari Pong task
  
  (Asked by: username_0 on 2019-01-25)
  
  I am in the process of implementing the DQN model from scratch in PyTorch with the target environment of Atari Pong. After a while of tweaking hyper-parameters, I cannot seem to get the model to achieve the performance that is reported in most publications (~ +21 reward; meaning that the agent wins almost every volley).
  
  My most recent results are shown in the following figure. Note that the x axis is episodes (full games to 21), but the total training iterations is ~6.7 million.
  
  [![enter image description here](https://i.sstatic.net/3TAEJ.png)](https://i.sstatic.net/3TAEJ.png)
  
  The specifics of my setup are as follows:
  
  ### Model
  
  ```
  class DQN(nn.Module):
      def __init__(self, in_channels, outputs):
          super(DQN, self).__init__()
          self.conv1 = nn.Conv2d(in_channels=in_channels, out_channels=32, kernel_size=8, stride=4)
          self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2)
          self.conv3 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1)
          self.fc1 = nn.Linear(in_features=64*7*7 , out_features=512)
          self.fc2 = nn.Linear(in_features=512, out_features=outputs)
  
      def forward(self, x):
          x = F.relu(self.conv1(x))
          x = F.relu(self.conv2(x))
          x = F.relu(self.conv3(x))
          x = x.view(-1, 64 * 7 * 7)
          x = F.relu(self.fc1(x))
          x = self.fc2(x)
          return x    # return Q values of each action
  
  ```
  
  ### Hyperparameters
  
  * batch size: 32
  * replay memory size: 100000
  * initial epsilon: 1.0
  * epsilon anneals linearly to 0.02 over 100000 steps
  * random warmstart episodes: ~50000
  * update target model every: 1000 steps
  * optimizer = optim.RMSprop(policy\_net.parameters(), lr=0.0025, alpha=0.9, eps=1e-02, momentum=0.0)
  
  ### Additional info
  
  * OpenAI gym Pong-v0 environment
  * Feeding model stacks of 4 last observed frames, scaled and cropped to 84x84 such that only the "playing area" is visible.
  * Treat losing a volley (end-of-life) as a terminal state in the replay buffer.
  * Using [smooth\_l1\_loss](https://pytorch.org/docs/stable/nn.html#smooth-l1-loss), which acts as Huber loss
  * Clipping gradients between -1 and 1 before optimizing
  * I offset the beginning of each episode with 4-30 no-op steps as the papers suggest
  
  Has anyone had a similar experience of getting stuck around 6 - 9 average reward per episode like this?
  
  Any suggestions for changes to hyperparameters or algorithmic nuances would be greatly appreciated!
  
  **Tags:** reinforcement-learning, deep-rl, dqn, implementation, atari-games
  
  > **Comments:**
  > **username_1**: Hi @username_0 , I am working on the same project right now, but my average score is capped at -10. Some differences I can see between our implementations are:  * "PongDeterministic-v4" environment which takes care of frame skipping. * did not crop the score part on the top, but my frames are 84 x 84. * do not treat losing a volley as a terminal state. * replay buffer size is 50000. RAM fills up and overflows if I use any more than that. No idea how you managed to store 100000 experience tuples. * using l2 loss, was planning to switch to huber loss though.  Do you have a Github repo?  
  > **username_0**: @username_1, to prevent memory overflow, I save the images as uint8 type in my replay buffer and cast (and divide by 255) right before I forward pass it to my model.  
  > **username_0**: By the way, the GitHub repo is https://github.com/MatthewInkawhich/learnRL  
  > **username_1**: I have done the same uint8 conversion. I was planning to normalize them by 255 too, but I believe they take up the same memory, whether divided or not. I was thinking I might lose information if I did that. What do you think? Also, I have followed you on github. In regards to your issue, I was thinking maybe it has something to do with your target network update frequency. It could be too low, causing instability during learning? Another thing you could try is a Prioritized Replay buffer.  
  > **username_0**: @username_1 thanks for the suggestions, I will try that. Also, I would recommend always normalizing your image inputs between 0 and 1, as it tends to play nicer with the initial weights and activation function.  
  > **username_1**: Thanks. I will probably implement that then. Along with better initialization schemes for the networks, gradient and reward clipping. You can check out my repo here if you like: https://github.com/username_1/Reinforcement-Learning/  
  > **username_1**: According to this answer [here](https://stackoverflow.com/a/38363141/3623131), using a squared loss function with gradient clipping is equivalent to using a Huber loss function without the gradient clipping. Gradient clipping step may not be required for you since you are using something close to Huber loss already.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-02-10. Score: 1
  
  Incase you still haven't been able to resolve the problem, here's a link to the [answer](https://ai.stackexchange.com/a/10481/21513) to my own question, which has the step-wise changes I made to achieve a +18 average score saturation using just a 10000 replay buffer size and a normal Double DQN (DDQN), trained for about 700-800 episodes. The updated code can also be found [here](https://github.com/username_1/Reinforcement-Learning/tree/master/Atari).
  
  No fancy changes like Prioritized Replay Buffer or any secret hyperparameter changes are required. It's usually something simple, like a small problem with the input preprocessing step.
  
  
  ---
  
  