Answers.Body: <p>It looks like on each step, you're calling both <code>self.model.predict</code> and <code>self.model.fit</code>. If you do this, you're going to run into stability problems, since your learning target is moving as you train.</p>
  
  <p>The way the DQN paper gets around this problem is by using 2 Q-networks, <span class="math-container">$Q$</span> and <span class="math-container">$\hat{Q}$</span>, where <span class="math-container">$\hat{Q}$</span> is called the target network. The target network's parameters are frozen, and its outputs are used to compute the learning targets for <span class="math-container">$Q$</span> (<code>targets_future</code> in your code). Every <span class="math-container">$C$</span> training steps (where <span class="math-container">$C$</span> is a hyperparameter), the target network <span class="math-container">$\hat{Q}$</span> is updated with the weights of <span class="math-container">$Q$</span>. See Algorithm 1 on Page 7 of the <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">DQN paper</a> for the details of this swap.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-30T21:04:51.740Z
Answers.Text: Could you check the edit I did in the original post? I'll appreciate it so much!
Answers.UserDisplayName: JCP
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-30T21:29:32.050Z
Answers.Text: When you're doing the Bellman update, you want to use `self.target_model.predict` rather than `self.model.predict`. Also, 100 steps seems low. You'll want to play around with that parameter but something >1000 is probably better.
Answers.UserDisplayName: Nishant Desai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-30T21:32:23.673Z
Answers.Text: So basically I only use the `model` CNN for the action update and the loss update am I right?
Answers.UserDisplayName: JCP
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-30T21:58:45.673Z
Answers.Text: That's right. `self.model` is the model that you call `.fit` on, and it's also the model you use to select your actions. There's a more detailed explanation of the difference between the two models on this question: https://ai.stackexchange.com/questions/6982/why-does-dqn-require-two-different-networks
Answers.UserDisplayName: Nishant Desai
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-01T02:36:14.967Z
Answers.Text: I still having a doubt. Should I update only the `target_model` weights every C training steps, or am I doing it right in the last edit I did? Thanks.
Answers.UserDisplayName: JCP
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-01T09:42:24.717Z
Answers.Text: @JCP I... think your **Edit 2** looks mostly fine. After the `if step_counter % 1000 == 0:`, you have a bunch of code duplicated that's also already running before that condition... this means you're running those duplicated parts twice in a row in the `step_count % 1000 == 0` case? That wouldn't be necessary. Just running all that code once, regardless of whether your step count is a multiple of `1000`, should be fine. Under the condition, you'll only need the new `agent.update_net()` call, which I assume is implemented to copy the weights of `self.model` into `self.target_model`?
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-30T19:35:48.240Z
Answers.LastActivityDate: 2018-11-30T19:35:48.240Z
Answers.UserDisplayName: Nishant Desai
Body: <p>I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. Each episode has a limit of 5000 steps but it couldn't reach more than 500-700. I think the problem is in the replay function, which is:</p>
  <pre><code>def replay(self, replay_batch_size, replay_batcher):
      j = 0
      k = 0
      replay_action = []
      replay_state = []
      replay_next_state = []
      replay_reward= []
      replay_superbatch = []
  
      if len(memory) &lt; replay_batch_size:
          replay_batch = random.sample(memory, len(memory))
          replay_batch = np.asarray(replay_batch)
          replay_state_batch, replay_next_state_batch, reward_batch, replay_action_batch = replay_batcher(replay_batch)
      else:
          replay_batch = random.sample(memory, replay_batch_size)
          replay_batch = np.asarray(replay_batch)
          replay_state_batch, replay_next_state_batch, reward_batch, replay_action_batch = replay_batcher(replay_batch)
          
      for j in range ((len(replay_batch)-len(replay_batch)%4)):
          
          if k &lt;= 4:
              k = k + 1              
              replay_state.append(replay_state_batch[j])
              replay_next_state.append(replay_next_state_batch[j])
              replay_reward.append(reward_batch[j])
              replay_action.append(replay_action_batch[j])
              
          if k &gt;=4:                
              k = 0
              replay_state = np.asarray(replay_state)
              replay_state.shape = shape
              replay_next_state = np.asarray(replay_next_state)
              replay_next_state.shape = shape
              replay_superbatch.append((replay_state, replay_next_state,replay_reward,replay_action))
  
              replay_state = []
              replay_next_state = []
              replay_reward = []
              replay_action = []
                                         
      states, target_future, targets_future, fit_batch = [], [], [], []
      
      for state_replay, next_state_replay, reward_replay, action_replay in replay_superbatch:
  
          target = reward_replay
          if not done:
              target = (reward_replay + self.gamma * np.amax(self.model.predict(next_state_replay)[0]))
  
          target_future = self.model.predict(state_replay)
  
          target_future[0][action_replay] = target
          states.append(state_replay[0])
          targets_future.append(target_future[0])
          fit_batch.append((states, targets_future))
  
      history = self.model.fit(np.asarray(states), np.array(targets_future), epochs=1, verbose=0)
  
      loss = history.history['loss'][0]
  
      if self.exploration_rate &gt; self.exploration_rate_min:
  
          self.exploration_rate -= (self.exploration_rate_decay/1000000)
      return loss
  </code></pre>
  <p>What I'm doing is to get 4 experiences (states), concatenate and introduce them in the CNN in shape (1, 210, 160, 4). Am I doing something wrong? If I implement the DDQN (Double Deep Q Net), should I obtain similar results as in the <a href="https://www.youtube.com/watch?v=V1eYniJ0Rnk" rel="nofollow noreferrer">DeepMind Breakout video</a>? Also, I'm using the Breakout-v0 enviroment from OpenAI gym.</p>
  <h3>Edit</h3>
  <p>Am I doing this properly? I implemented an identical CNN; then I update the target each 100 steps and copy the weights from <code>model</code> CNN  to <code>target_model</code> CNN. Should it improve the learning? Anyway I'm getting low loss.</p>
  <pre><code>for state_replay, next_state_replay, reward_replay, action_replay in replay_superbatch:
  
              target = reward_replay
              if not done:
  
                  target = (reward_replay + self.gamma * np.amax(self.model.predict(next_state_replay)[0]))
              if steps % 100 == 0:
  
                  target_future = self.target_model.predict(state_replay)
  
                  target_future[0][action_replay] = target
                  states.append(state_replay[0])
                  targets_future.append(target_future[0])
                  fit_batch.append((states, targets_future))
                  agent.update_net()
  
          history = self.model.fit(np.asarray(states), np.array(targets_future), epochs=1, verbose=0)
  
          loss = history.history['loss'][0]
  </code></pre>
  <h3>Edit 2</h3>
  <p>So as far I understand, this code should work am I right?</p>
  <pre><code>if not done:
  
              target = (reward_replay + self.gamma * np.amax(self.target_model.predict(next_state_replay)[0]))
              target.shape = (1,4)
              
              target[0][action_replay] = target
              target_future = target
              states.append(state_replay[0])
              targets_future.append(target_future[0])
              fit_batch.append((states, targets_future))
  
          if step_counter % 1000 == 0:
  
              target_future = self.target_model.predict(state_replay)
  
              target_future[0][action_replay] = target
              states.append(state_replay[0])
              targets_future.append(target_future[0])
              fit_batch.append((states, targets_future))
              agent.update_net()
  
      history = self.model.fit(np.asarray(states), np.array(targets_future), epochs=1, verbose=0)
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-26T06:46:24.523Z
Comments.Text: DQN needs a lot of episodes. Additionally, the training is super unstable. Meaning out of 100 training runs it could happen that 95 are completely unusable and 5 are fine.
Comments.UserDisplayName: Martin Thoma
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-27T04:32:18.423Z
Comments.Text: So, how can I improve the training stability? Double DQN should improve it?
Comments.UserDisplayName: JCP
Tags: <reinforcement-learning><deep-rl><q-learning><dqn><implementation>
Title: Why isn't my DQN implementation working properly?
UserDisplayName: JCP
ThreadText: # Why isn't my DQN implementation working properly?
  
  (Asked by: username_0 on 2018-11-26)
  
  I'm trying to build a DQN to replicate the DeepMind results. I'm doing with a simple DQN for the moment, but it isn't learning properly: after +5000 episodes, it couldn't get more than 9-10 points. Each episode has a limit of 5000 steps but it couldn't reach more than 500-700. I think the problem is in the replay function, which is:
  def replay(self, replay_batch_size, replay_batcher):
      j = 0
      k = 0
      replay_action = []
      replay_state = []
      replay_next_state = []
      replay_reward= []
      replay_superbatch = []
  
      if len(memory) < replay_batch_size:
          replay_batch = random.sample(memory, len(memory))
          replay_batch = np.asarray(replay_batch)
          replay_state_batch, replay_next_state_batch, reward_batch, replay_action_batch = replay_batcher(replay_batch)
      else:
          replay_batch = random.sample(memory, replay_batch_size)
          replay_batch = np.asarray(replay_batch)
          replay_state_batch, replay_next_state_batch, reward_batch, replay_action_batch = replay_batcher(replay_batch)
          
      for j in range ((len(replay_batch)-len(replay_batch)%4)):
          
          if k <= 4:
              k = k + 1              
              replay_state.append(replay_state_batch[j])
              replay_next_state.append(replay_next_state_batch[j])
              replay_reward.append(reward_batch[j])
              replay_action.append(replay_action_batch[j])
              
          if k >=4:                
              k = 0
              replay_state = np.asarray(replay_state)
              replay_state.shape = shape
              replay_next_state = np.asarray(replay_next_state)
              replay_next_state.shape = shape
              replay_superbatch.append((replay_state, replay_next_state,replay_reward,replay_action))
  
              replay_state = []
              replay_next_state = []
              replay_reward = []
              replay_action = []
                                         
      states, target_future, targets_future, fit_batch = [], [], [], []
      
      for state_replay, next_state_replay, reward_replay, action_replay in replay_superbatch:
  
          target = reward_replay
          if not done:
              target = (reward_replay + self.gamma * np.amax(self.model.predict(next_state_replay)[0]))
  
          target_future = self.model.predict(state_replay)
  
          target_future[0][action_replay] = target
          states.append(state_replay[0])
          targets_future.append(target_future[0])
          fit_batch.append((states, targets_future))
  
      history = self.model.fit(np.asarray(states), np.array(targets_future), epochs=1, verbose=0)
  
      loss = history.history['loss'][0]
  
      if self.exploration_rate > self.exploration_rate_min:
  
          self.exploration_rate -= (self.exploration_rate_decay/1000000)
      return loss
  
  What I'm doing is to get 4 experiences (states), concatenate and introduce them in the CNN in shape (1, 210, 160, 4). Am I doing something wrong? If I implement the DDQN (Double Deep Q Net), should I obtain similar results as in the DeepMind Breakout video? Also, I'm using the Breakout-v0 enviroment from OpenAI gym.
  Edit
  Am I doing this properly? I implemented an identical CNN; then I update the target each 100 steps and copy the weights from model CNN  to target_model CNN. Should it improve the learning? Anyway I'm getting low loss.
  for state_replay, next_state_replay, reward_replay, action_replay in replay_superbatch:
  
              target = reward_replay
              if not done:
  
                  target = (reward_replay + self.gamma * np.amax(self.model.predict(next_state_replay)[0]))
              if steps % 100 == 0:
  
                  target_future = self.target_model.predict(state_replay)
  
                  target_future[0][action_replay] = target
                  states.append(state_replay[0])
                  targets_future.append(target_future[0])
                  fit_batch.append((states, targets_future))
                  agent.update_net()
  
          history = self.model.fit(np.asarray(states), np.array(targets_future), epochs=1, verbose=0)
  
          loss = history.history['loss'][0]
  
  Edit 2
  So as far I understand, this code should work am I right?
  if not done:
  
              target = (reward_replay + self.gamma * np.amax(self.target_model.predict(next_state_replay)[0]))
              target.shape = (1,4)
              
              target[0][action_replay] = target
              target_future = target
              states.append(state_replay[0])
              targets_future.append(target_future[0])
              fit_batch.append((states, targets_future))
  
          if step_counter % 1000 == 0:
  
              target_future = self.target_model.predict(state_replay)
  
              target_future[0][action_replay] = target
              states.append(state_replay[0])
              targets_future.append(target_future[0])
              fit_batch.append((states, targets_future))
              agent.update_net()
  
      history = self.model.fit(np.asarray(states), np.array(targets_future), epochs=1, verbose=0)
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, dqn, implementation
  > **Comments:**
  > **username_1**: DQN needs a lot of episodes. Additionally, the training is super unstable. Meaning out of 100 training runs it could happen that 95 are completely unusable and 5 are fine.  
  > **username_0**: So, how can I improve the training stability? Double DQN should improve it?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-11-30. Score: 2 | Accepted answer
  
  It looks like on each step, you're calling both `self.model.predict` and `self.model.fit`. If you do this, you're going to run into stability problems, since your learning target is moving as you train.
  
  The way the DQN paper gets around this problem is by using 2 Q-networks, $Q$ and $\hat{Q}$, where $\hat{Q}$ is called the target network. The target network's parameters are frozen, and its outputs are used to compute the learning targets for $Q$ (`targets_future` in your code). Every $C$ training steps (where $C$ is a hyperparameter), the target network $\hat{Q}$ is updated with the weights of $Q$. See Algorithm 1 on Page 7 of the [DQN paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf) for the details of this swap.
  
  > **Comments:**
  > **username_0**: Could you check the edit I did in the original post? I'll appreciate it so much!  
  > **username_2**: When you're doing the Bellman update, you want to use `self.target_model.predict` rather than `self.model.predict`. Also, 100 steps seems low. You'll want to play around with that parameter but something >1000 is probably better.  
  > **username_0**: So basically I only use the `model` CNN for the action update and the loss update am I right?  
  > **username_2**: That's right. `self.model` is the model that you call `.fit` on, and it's also the model you use to select your actions. There's a more detailed explanation of the difference between the two models on this question: https://ai.stackexchange.com/questions/6982/why-does-dqn-require-two-different-networks  
  > **username_0**: I still having a doubt. Should I update only the `target_model` weights every C training steps, or am I doing it right in the last edit I did? Thanks.  
  > **username_3**: @JCP I... think your **Edit 2** looks mostly fine. After the `if step_counter % 1000 == 0:`, you have a bunch of code duplicated that's also already running before that condition... this means you're running those duplicated parts twice in a row in the `step_count % 1000 == 0` case? That wouldn't be necessary. Just running all that code once, regardless of whether your step count is a multiple of `1000`, should be fine. Under the condition, you'll only need the new `agent.update_net()` call, which I assume is implemented to copy the weights of `self.model` into `self.target_model`?  
  
  ---
  
  