Answers.Body: <p>First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T12:26:28.863Z
Answers.Text: Td_target is the estimation of our current state's but it q value is base on next state.
Answers.UserDisplayName: robot tech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T12:49:41.847Z
Answers.Text: In DQN, the td_target is the optimal q value and you are subtracting it from your estimated q value.
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T12:53:51.073Z
Answers.Text: As i mentioned above, td_targets is tf.stop_gradient(rewards + discounts * next_q_values). It is not just simple next_q_values.
Answers.UserDisplayName: robot tech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T13:25:57.000Z
Answers.Text: In DQN, td_targets = rewards + discount*(max q value of next state). The max term comes from the target policy.  If the explanation did not satisfy you, can you please explain to me your question in more detail(without adding the code section)?
Answers.UserDisplayName: Swakshar Deb
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T12:32:12.690Z
Answers.Text: I am just do not understand how come it working. Maybe i need more this kind of lost solution on other senario. Just for appreciation and have intuition.
Answers.UserDisplayName: robot tech
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-26T12:15:01.723Z
Answers.LastActivityDate: 2020-04-26T12:27:37.707Z
Answers.LastEditDate: 2020-04-26T12:27:37.707Z
Answers.UserDisplayName: Swakshar Deb
Body: <p>Why <code>td_loss</code> is calculated from (<code>td_targets</code> against <code>q_values</code>)?</p>
  
  <p>Why I am lost is because:</p>
  
  <ol>
  <li><code>q_values</code> is just the probability of action. It does not have a reward and discount.</li>
  <li><code>td_targets</code> does have rewards + discounts * next_q_values. Somemore next_q_values is next state.</li>
  </ol>
  
  <p>How both <code>td_targets</code> and <code>q_values</code> can minus (or Huber or MSE) to get lost work?</p>
  
  <pre><code>td_error = valid_mask * (td_targets - q_values)
  td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)
  </code></pre>
  
  <p><a href="https://github.com/tensorflow/agents/blob/v0.4.0/tf_agents/agents/dqn/dqn_agent.py#L459" rel="nofollow noreferrer">td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)</a></p>
  
Tags: <objective-functions><dqn>
Title: How does the DQN loss from td_targets against q_values make sense?
UserDisplayName: robot tech
ThreadText: # How does the DQN loss from td_targets against q_values make sense?
  
  (Asked by: username_0 on 2020-04-26)
  
  Why td_loss is calculated from (td_targets against q_values)?
  Why I am lost is because:
  
  q_values is just the probability of action. It does not have a reward and discount.
  td_targets does have rewards + discounts * next_q_values. Somemore next_q_values is next state.
  
  How both td_targets and q_values can minus (or Huber or MSE) to get lost work?
  td_error = valid_mask * (td_targets - q_values)
  td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)
  
  td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)
  
  **Tags:** objective-functions, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-26. Score: -1
  
  First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td\_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q\_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.
  
  > **Comments:**
  > **username_0**: Td_target is the estimation of our current state's but it q value is base on next state.  
  > **username_1**: In DQN, the td_target is the optimal q value and you are subtracting it from your estimated q value.  
  > **username_0**: As i mentioned above, td_targets is tf.stop_gradient(rewards + discounts * next_q_values). It is not just simple next_q_values.  
  > **username_1**: In DQN, td_targets = rewards + discount*(max q value of next state). The max term comes from the target policy.  If the explanation did not satisfy you, can you please explain to me your question in more detail(without adding the code section)?  
  > **username_0**: I am just do not understand how come it working. Maybe i need more this kind of lost solution on other senario. Just for appreciation and have intuition.  
  
  ---
  
  