Answers.Body: <p>The loss function is designed in a way to approximate the bellman optimality for <span class="math-container">$Q^*(s,a)$</span>. Given an optimal policy <span class="math-container">$\pi^*$</span>, <span class="math-container">$Q^*(s,a)$</span> satisfies the equation <span class="math-container">$$Q^*(s,a) = r(s) + \gamma max_{a'}\sum_{s'}P(s'|s,a)Q^*(s',a')$$</span></p>
  <p>At convergence, the highest <span class="math-container">$Q$</span> value that I can get taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> is equal to the reward I get for taking action <span class="math-container">$a$</span> and the max <span class="math-container">$Q$</span> values at the next state.</p>
  <p>You can see in the loss function that the DQN tries to attain a <span class="math-container">$Q^*(s,a)$</span> value that closely approximates the equation above.</p>
  <p>On a side note, in model based RL setting, the bellman optimality for <span class="math-container">$Q^*$</span> is expressed as <span class="math-container">$$Q^*(s,a) = r(s) + \gamma \sum_{s'}P(s'|s,a)V^*(s)$$</span> <span class="math-container">$V^*(s')$</span> is used in the model based case because <span class="math-container">$V^*(s)$</span> by definition represents the highest possible value attainable at state <span class="math-container">$s'$</span> following <span class="math-container">$\pi^*$</span>. In model free setting, <span class="math-container">$V^*$</span> is replaced by <span class="math-container">$Q^*$</span> because computing <span class="math-container">$V^*$</span> is not useful in achieving model free control without a transition model <span class="math-container">$P(s'|s,a)$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T07:41:15.973Z
Answers.Text: OP is asking about use of target network for setting TD target (as opposed to the "current" Q function approximation)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T03:45:12.613Z
Answers.LastActivityDate: 2020-11-12T03:45:12.613Z
Answers.UserDisplayName: calveeen
Body: <p>I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss between the <code>qVal</code> for the next state in our target network and the <strong>current</strong> state in the local network. The Nature paper by Mnih et al. is well explained, however, I am not getting from it the purpose of the above. Here is my training portion from a script I am running:</p>
  <pre><code>for state, action, reward, next_state, done in minibatch:
      target_qVal = self.model.predict(state)
  
      # print(target_qVal)
  
      if done:
          target_qVal[0][action] = reward #done
      else:
          # predicted q value for next state from target model
          pred = self.target_model.predict(next_state)[0]
          target_qVal[0][action] = reward + self.gamma * np.amax(pred)
  
      # indentation position?
      self.model.fit(np.array(state), 
                     np.array(target_qVal), 
                     batch_size=batch_size,
                     verbose=0, 
                     shuffle=False, 
                     epochs=1)
  </code></pre>
  <p>I understand that the expected return is the immediate reward plus the cumulative sum of discounted rewards looking into the future <span class="math-container">$s'$</span> (correct me if I'm wrong in my understanding) when following a given policy.</p>
  <p>My fundamental misunderstanding is the loss equation:</p>
  <p><span class="math-container">$$L = [r + \gamma \max Q(s',a'; \theta') - Q(s,a; \theta)],$$</span></p>
  <p>where <span class="math-container">$\theta'$</span> and <span class="math-container">$\theta$</span> are the weights of the target and online neural networks, respectively.</p>
  <p><em>Why do we aim to minimize the Q value of the next state in the target model and the Q value of the current state in the online model?</em></p>
  <p>A bonus question would be, in order to collect <span class="math-container">$Q(s,a)$</span> values for dimensionality reduction (as in Mnih et al t-sne plot), would I simply collect the <code>target_qVal[0]</code> values during training and feed them into a list after each step to accumulate the Q values over time?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-11T20:05:52.573Z
Comments.Text: could you please edit your question so the loss function is written using LaTeX as at the moment it is unreadable.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T10:47:27.570Z
Comments.Text: I have edited your post to fix some formatting issues, etc. Please, make sure that it is still consistent with your original post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T11:04:26.853Z
Comments.Text: thank you! apologies for not correctly entewring the equation initially. this is correct and does convey my question around why it is the Q of s' and a' minus q val of current s and a.
Comments.UserDisplayName: user9317212
Tags: <reinforcement-learning><dqn><deep-rl><double-dqn>
Title: Why do we minimise the loss between the target Q values and 'local' Q values?
UserDisplayName: user9317212
ThreadText: # Why do we minimise the loss between the target Q values and 'local' Q values?
  
  (Asked by: username_0 on 2020-11-11)
  
  I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss between the `qVal` for the next state in our target network and the **current** state in the local network. The Nature paper by Mnih et al. is well explained, however, I am not getting from it the purpose of the above. Here is my training portion from a script I am running:
  
  ```
  for state, action, reward, next_state, done in minibatch:
      target_qVal = self.model.predict(state)
  
      # print(target_qVal)
  
      if done:
          target_qVal[0][action] = reward #done
      else:
          # predicted q value for next state from target model
          pred = self.target_model.predict(next_state)[0]
          target_qVal[0][action] = reward + self.gamma * np.amax(pred)
  
      # indentation position?
      self.model.fit(np.array(state), 
                     np.array(target_qVal), 
                     batch_size=batch_size,
                     verbose=0, 
                     shuffle=False, 
                     epochs=1)
  
  ```
  
  I understand that the expected return is the immediate reward plus the cumulative sum of discounted rewards looking into the future $s'$ (correct me if I'm wrong in my understanding) when following a given policy.
  
  My fundamental misunderstanding is the loss equation:
  
  $$L = [r + \gamma \max Q(s',a'; \theta') - Q(s,a; \theta)],$$
  
  where $\theta'$ and $\theta$ are the weights of the target and online neural networks, respectively.
  
  *Why do we aim to minimize the Q value of the next state in the target model and the Q value of the current state in the online model?*
  
  A bonus question would be, in order to collect $Q(s,a)$ values for dimensionality reduction (as in Mnih et al t-sne plot), would I simply collect the `target_qVal[0]` values during training and feed them into a list after each step to accumulate the Q values over time?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, double-dqn
  
  > **Comments:**
  > **username_1**: could you please edit your question so the loss function is written using LaTeX as at the moment it is unreadable.  
  > **username_2**: I have edited your post to fix some formatting issues, etc. Please, make sure that it is still consistent with your original post.  
  > **username_0**: thank you! apologies for not correctly entewring the equation initially. this is correct and does convey my question around why it is the Q of s' and a' minus q val of current s and a.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-11-12. Score: 1
  
  The loss function is designed in a way to approximate the bellman optimality for $Q^\*(s,a)$. Given an optimal policy $\pi^\*$, $Q^\*(s,a)$ satisfies the equation $$Q^\*(s,a) = r(s) + \gamma max\_{a'}\sum\_{s'}P(s'|s,a)Q^\*(s',a')$$
  
  At convergence, the highest $Q$ value that I can get taking action $a$ in state $s$ is equal to the reward I get for taking action $a$ and the max $Q$ values at the next state.
  
  You can see in the loss function that the DQN tries to attain a $Q^\*(s,a)$ value that closely approximates the equation above.
  
  On a side note, in model based RL setting, the bellman optimality for $Q^\*$ is expressed as $$Q^\*(s,a) = r(s) + \gamma \sum\_{s'}P(s'|s,a)V^\*(s)$$ $V^\*(s')$ is used in the model based case because $V^\*(s)$ by definition represents the highest possible value attainable at state $s'$ following $\pi^\*$. In model free setting, $V^\*$ is replaced by $Q^\*$ because computing $V^\*$ is not useful in achieving model free control without a transition model $P(s'|s,a)$
  
  > **Comments:**
  > **username_4**: OP is asking about use of target network for setting TD target (as opposed to the "current" Q function approximation)  
  
  ---
  
  