Answers.Body: <p>The loss function is designed in a way to approximate the bellman optimality for <span class="math-container">$Q^*(s,a)$</span>. Given an optimal policy <span class="math-container">$\pi^*$</span>, <span class="math-container">$Q^*(s,a)$</span> satisfies the equation <span class="math-container">$$Q^*(s,a) = r(s) + \gamma max_{a'}\sum_{s'}P(s'|s,a)Q^*(s',a')$$</span></p>
  <p>At convergence, the highest <span class="math-container">$Q$</span> value that I can get taking action <span class="math-container">$a$</span> in state <span class="math-container">$s$</span> is equal to the reward I get for taking action <span class="math-container">$a$</span> and the max <span class="math-container">$Q$</span> values at the next state.</p>
  <p>You can see in the loss function that the DQN tries to attain a <span class="math-container">$Q^*(s,a)$</span> value that closely approximates the equation above.</p>
  <p>On a side note, in model based RL setting, the bellman optimality for <span class="math-container">$Q^*$</span> is expressed as <span class="math-container">$$Q^*(s,a) = r(s) + \gamma \sum_{s'}P(s'|s,a)V^*(s)$$</span> <span class="math-container">$V^*(s')$</span> is used in the model based case because <span class="math-container">$V^*(s)$</span> by definition represents the highest possible value attainable at state <span class="math-container">$s'$</span> following <span class="math-container">$\pi^*$</span>. In model free setting, <span class="math-container">$V^*$</span> is replaced by <span class="math-container">$Q^*$</span> because computing <span class="math-container">$V^*$</span> is not useful in achieving model free control without a transition model <span class="math-container">$P(s'|s,a)$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T07:41:15.973Z
Answers.Text: OP is asking about use of target network for setting TD target (as opposed to the "current" Q function approximation)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-12T03:45:12.613Z
Answers.LastActivityDate: 2020-11-12T03:45:12.613Z
Answers.UserDisplayName: calveeen
Body: <p>I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss between the <code>qVal</code> for the next state in our target network and the <strong>current</strong> state in the local network. The Nature paper by Mnih et al. is well explained, however, I am not getting from it the purpose of the above. Here is my training portion from a script I am running:</p>
  <pre><code>for state, action, reward, next_state, done in minibatch:
      target_qVal = self.model.predict(state)
  
      # print(target_qVal)
  
      if done:
          target_qVal[0][action] = reward #done
      else:
          # predicted q value for next state from target model
          pred = self.target_model.predict(next_state)[0]
          target_qVal[0][action] = reward + self.gamma * np.amax(pred)
  
      # indentation position?
      self.model.fit(np.array(state), 
                     np.array(target_qVal), 
                     batch_size=batch_size,
                     verbose=0, 
                     shuffle=False, 
                     epochs=1)
  </code></pre>
  <p>I understand that the expected return is the immediate reward plus the cumulative sum of discounted rewards looking into the future <span class="math-container">$s'$</span> (correct me if I'm wrong in my understanding) when following a given policy.</p>
  <p>My fundamental misunderstanding is the loss equation:</p>
  <p><span class="math-container">$$L = [r + \gamma \max Q(s',a'; \theta') - Q(s,a; \theta)],$$</span></p>
  <p>where <span class="math-container">$\theta'$</span> and <span class="math-container">$\theta$</span> are the weights of the target and online neural networks, respectively.</p>
  <p><em>Why do we aim to minimize the Q value of the next state in the target model and the Q value of the current state in the online model?</em></p>
  <p>A bonus question would be, in order to collect <span class="math-container">$Q(s,a)$</span> values for dimensionality reduction (as in Mnih et al t-sne plot), would I simply collect the <code>target_qVal[0]</code> values during training and feed them into a list after each step to accumulate the Q values over time?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-11T20:05:52.573Z
Comments.Text: could you please edit your question so the loss function is written using LaTeX as at the moment it is unreadable.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T10:47:27.570Z
Comments.Text: I have edited your post to fix some formatting issues, etc. Please, make sure that it is still consistent with your original post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T11:04:26.853Z
Comments.Text: thank you! apologies for not correctly entewring the equation initially. this is correct and does convey my question around why it is the Q of s' and a' minus q val of current s and a.
Comments.UserDisplayName: user9317212
Tags: <reinforcement-learning><dqn><deep-rl><double-dqn>
Title: Why do we minimise the loss between the target Q values and 'local' Q values?
UserDisplayName: user9317212
ThreadText: # Why do we minimise the loss between the target Q values and 'local' Q values?
  
  (Asked by: username_0 on 2020-11-11)
  
  I have a question regarding the loss function of target networks and current (online) networks. I understand the action value function. What I am unsure about is why we seek to minimise the loss between the qVal for the next state in our target network and the current state in the local network. The Nature paper by Mnih et al. is well explained, however, I am not getting from it the purpose of the above. Here is my training portion from a script I am running:
  for state, action, reward, next_state, done in minibatch:
      target_qVal = self.model.predict(state)
  
      # print(target_qVal)
  
      if done:
          target_qVal[0][action] = reward #done
      else:
          # predicted q value for next state from target model
          pred = self.target_model.predict(next_state)[0]
          target_qVal[0][action] = reward + self.gamma * np.amax(pred)
  
      # indentation position?
      self.model.fit(np.array(state), 
                     np.array(target_qVal), 
                     batch_size=batch_size,
                     verbose=0, 
                     shuffle=False, 
                     epochs=1)
  
  I understand that the expected return is the immediate reward plus the cumulative sum of discounted rewards looking into the future $s'$ (correct me if I'm wrong in my understanding) when following a given policy.
  My fundamental misunderstanding is the loss equation:
  $$L = [r + \gamma \max Q(s',a'; \theta') - Q(s,a; \theta)],$$
  where $\theta'$ and $\theta$ are the weights of the target and online neural networks, respectively.
  Why do we aim to minimize the Q value of the next state in the target model and the Q value of the current state in the online model?
  A bonus question would be, in order to collect $Q(s,a)$ values for dimensionality reduction (as in Mnih et al t-sne plot), would I simply collect the target_qVal[0] values during training and feed them into a list after each step to accumulate the Q values over time?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, double-dqn
  > **Comments:**
  > **username_1**: could you please edit your question so the loss function is written using LaTeX as at the moment it is unreadable.  
  > **username_2**: I have edited your post to fix some formatting issues, etc. Please, make sure that it is still consistent with your original post.  
  > **username_0**: thank you! apologies for not correctly entewring the equation initially. this is correct and does convey my question around why it is the Q of s' and a' minus q val of current s and a.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-11-12. Score: 1
  
  The loss function is designed in a way to approximate the bellman optimality for $Q^\*(s,a)$. Given an optimal policy $\pi^\*$, $Q^\*(s,a)$ satisfies the equation $$Q^\*(s,a) = r(s) + \gamma max\_{a'}\sum\_{s'}P(s'|s,a)Q^\*(s',a')$$
  
  At convergence, the highest $Q$ value that I can get taking action $a$ in state $s$ is equal to the reward I get for taking action $a$ and the max $Q$ values at the next state.
  
  You can see in the loss function that the DQN tries to attain a $Q^\*(s,a)$ value that closely approximates the equation above.
  
  On a side note, in model based RL setting, the bellman optimality for $Q^\*$ is expressed as $$Q^\*(s,a) = r(s) + \gamma \sum\_{s'}P(s'|s,a)V^\*(s)$$ $V^\*(s')$ is used in the model based case because $V^\*(s)$ by definition represents the highest possible value attainable at state $s'$ following $\pi^\*$. In model free setting, $V^\*$ is replaced by $Q^\*$ because computing $V^\*$ is not useful in achieving model free control without a transition model $P(s'|s,a)$
  
  > **Comments:**
  > **username_4**: OP is asking about use of target network for setting TD target (as opposed to the "current" Q function approximation)  
  
  ---
  
  