Answers.Body: <p>In my experience, neural networks with convolutional layers take much much longer to train, so try increasing the number of iterations (time steps). After running, save the network model (I dont know how to do it in torch, but in tensorflow it was model.save(&quot;filename&quot;+&quot;.h5&quot;) ).</p>
  <p>Then, load this saved model file and do a test run to see if it worked. In this case, you should notice pretty easily if it learned or not).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T13:39:54.087Z
Answers.LastActivityDate: 2021-03-18T13:39:54.087Z
Answers.UserDisplayName: Lucas Pelizzari
Answers.Body: <p>Since you are looking at a single iteration and expect a meaningful change my guess is that you aren't training for long enough. Q-learning can take very long, for many environments it takes millions of iterations.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T16:30:36.467Z
Answers.LastActivityDate: 2020-10-18T16:30:36.467Z
Answers.UserDisplayName: Tom Dörr
Answers.Body: <p>I found the reason it wasn't learning. The issue was this line of code:</p>
  <pre><code>q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()
  </code></pre>
  <p>I had been using the tilde operator before to invert uint8 tensors, but recently I had updated to the latest version of pytorch that seems to have changed how the operator works. It was changing the done values to 255.</p>
  <p>Changing to this line fixed it:</p>
  <pre><code>q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (1 - dones)
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-28T18:32:55.563Z
Answers.LastActivityDate: 2021-03-28T18:32:55.563Z
Answers.UserDisplayName: Ryan McCauley
Body: <p>I am trying to create a simple Deep Q-Network with 2d convolutional layers.</p>
  <p>I can't figure out what I am doing wrong, and the only thing I can see that doesn't seem right is when I get the model prediction for a state after the optimizer step it doesn’t seem to get closer to the target.</p>
  <p>I am using pixels from pong in OpenAI's gym with single-channel 90x90 images, a batch size of 32, and replay memory.</p>
  <p>As an example, if I try with a batch size of 1, and try running <code>self(states)</code> again right after the optimizer step the output is as follows:</p>
  <pre><code>current_q_values -&gt; -0.16351485  0.29163417  0.11192469 -0.08969332  0.11081569  0.37215832
  q_target -&gt;         -0.16351485  0.5336551   0.11192469 -0.08969332  0.11081569  0.37215832
  self(states) -&gt;     -0.8427617   0.6415581   0.44988257 -0.43897176  0.8693738   0.40007943
  </code></pre>
  <p>Does this look as what would be expected for a single step?</p>
  <p>The network with loss and optimizer:</p>
  <pre><code>    self.in_layer = Conv2d(channels, 32, 8)
      self.hidden_conv_1 = Conv2d(32, 64, 4)
      self.hidden_conv_2 = Conv2d(64, 128, 3)
      self.hidden_fc1 = Linear(128 * 78 * 78, 64)
      self.hidden_fc2 = Linear(64, 32)
      self.output = Linear(32, action_space)
  
      self.loss = torch.nn.MSELoss()
      self.optimizer = torch.optim.Adam(
          self.parameters(), lr=learning_rate) # lr is 0.001
  
  def forward(self, state):
      in_out = fn.relu(self.in_layer(state))
      in_out = fn.relu(self.hidden_conv_1(in_out))
      in_out = fn.relu(self.hidden_conv_2(in_out))
      in_out = in_out.view(-1, 128 * 78 * 78)
      in_out = fn.relu(self.hidden_fc1(in_out))
      in_out = fn.relu(self.hidden_fc2(in_out))
      return self.output(in_out)
  </code></pre>
  <p>Then the learning block:</p>
  <pre><code>        self.optimizer.zero_grad()
  
          sample = self.sample(self.batch_size)
          states = torch.stack([i[0] for i in sample])
          actions = torch.tensor([i[1] for i in sample], device=device)
          rewards = torch.tensor([i[2] for i in sample], dtype=torch.float32, device=device)
          next_states = torch.stack([i[3] for i in sample])
          dones = torch.tensor([i[4] for i in sample], dtype=torch.uint8, device=device)
  
          current_q_vals = self(states)
          next_q_vals = self(next_states)
          q_target = current_q_vals.clone()
          q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()
  
          loss = fn.smooth_l1_loss(current_q_vals, q_target)
          loss.backward()
  
          self.optimizer.step()
  <span class="math-container">```</span>
  </code></pre>
  
Tags: <reinforcement-learning><q-learning><dqn><pytorch><convolutional-layers>
Title: DQN not learning and step not stepping towards target
UserDisplayName: Ryan McCauley
ThreadText: # DQN not learning and step not stepping towards target
  
  (Asked by: username_0 on 2020-09-20)
  
  I am trying to create a simple Deep Q-Network with 2d convolutional layers.
  I can't figure out what I am doing wrong, and the only thing I can see that doesn't seem right is when I get the model prediction for a state after the optimizer step it doesn’t seem to get closer to the target.
  I am using pixels from pong in OpenAI's gym with single-channel 90x90 images, a batch size of 32, and replay memory.
  As an example, if I try with a batch size of 1, and try running self(states) again right after the optimizer step the output is as follows:
  current_q_values -> -0.16351485  0.29163417  0.11192469 -0.08969332  0.11081569  0.37215832
  q_target ->         -0.16351485  0.5336551   0.11192469 -0.08969332  0.11081569  0.37215832
  self(states) ->     -0.8427617   0.6415581   0.44988257 -0.43897176  0.8693738   0.40007943
  
  Does this look as what would be expected for a single step?
  The network with loss and optimizer:
      self.in_layer = Conv2d(channels, 32, 8)
      self.hidden_conv_1 = Conv2d(32, 64, 4)
      self.hidden_conv_2 = Conv2d(64, 128, 3)
      self.hidden_fc1 = Linear(128 * 78 * 78, 64)
      self.hidden_fc2 = Linear(64, 32)
      self.output = Linear(32, action_space)
  
      self.loss = torch.nn.MSELoss()
      self.optimizer = torch.optim.Adam(
          self.parameters(), lr=learning_rate) # lr is 0.001
  
  def forward(self, state):
      in_out = fn.relu(self.in_layer(state))
      in_out = fn.relu(self.hidden_conv_1(in_out))
      in_out = fn.relu(self.hidden_conv_2(in_out))
      in_out = in_out.view(-1, 128 * 78 * 78)
      in_out = fn.relu(self.hidden_fc1(in_out))
      in_out = fn.relu(self.hidden_fc2(in_out))
      return self.output(in_out)
  
  Then the learning block:
          self.optimizer.zero_grad()
  
          sample = self.sample(self.batch_size)
          states = torch.stack([i[0] for i in sample])
          actions = torch.tensor([i[1] for i in sample], device=device)
          rewards = torch.tensor([i[2] for i in sample], dtype=torch.float32, device=device)
          next_states = torch.stack([i[3] for i in sample])
          dones = torch.tensor([i[4] for i in sample], dtype=torch.uint8, device=device)
  
          current_q_vals = self(states)
          next_q_vals = self(next_states)
          q_target = current_q_vals.clone()
          q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()
  
          loss = fn.smooth_l1_loss(current_q_vals, q_target)
          loss.backward()
  
          self.optimizer.step()
  ```
  
  **Tags:** reinforcement-learning, q-learning, dqn, pytorch, convolutional-layers
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2021-03-18. Score: 1
  
  In my experience, neural networks with convolutional layers take much much longer to train, so try increasing the number of iterations (time steps). After running, save the network model (I dont know how to do it in torch, but in tensorflow it was model.save("filename"+".h5") ).
  
  Then, load this saved model file and do a test run to see if it worked. In this case, you should notice pretty easily if it learned or not).
  
  
  ---
  
  ### Answer by username_2 on 2020-10-18. Score: 0
  
  Since you are looking at a single iteration and expect a meaningful change my guess is that you aren't training for long enough. Q-learning can take very long, for many environments it takes millions of iterations.
  
  
  ---
  
  ### Answer by username_0 on 2021-03-28. Score: 0
  
  I found the reason it wasn't learning. The issue was this line of code:
  q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float()
  
  I had been using the tilde operator before to invert uint8 tensors, but recently I had updated to the latest version of pytorch that seems to have changed how the operator works. It was changing the done values to 255.
  Changing to this line fixed it:
  q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (1 - dones)
  
  
  ---
  
  