Answers.Body: <p>The labels in DQN, and in Q-learning in general, are <em>not</em> &quot;true&quot; in the sense that they represent optimal action value functions. Instead they represent approximate action values of a <em>current</em> target policy.</p>
  <p>The target policy changes every C time steps, when the network with static weights is updated. This update will include both corrections to the action value approximations, and changes to which actions are considered optimal.</p>
  <p>The reason this converges towards an optimal action value function is related to the <a href="http://incompleteideas.net/book/ebook/node42.html" rel="nofollow noreferrer">policy improvement theorem</a>. With function approximation, as in DQN, the convergence is not guaranteed, but the process is stll based on the same idea. In summary it is a two-step repeated feedback process:</p>
  <ol>
  <li>learn value function of a current policy</li>
  <li>update the policy to select actions with maximum values</li>
  </ol>
  <p>What this means for the TD target &quot;labels&quot; in DQN is:</p>
  <ul>
  <li><p>They are not ground truth for the optimal action value function, until <em>after</em> the algorithm has converged.</p>
  </li>
  <li><p>They are biased, initially almost completely arbitrarily by however the target network has been initialised, and from then on due to a slowly-reducing impact from that initial bias and from lagging behind collected data.</p>
  </li>
  <li><p>They are non-stationary. This means an online learning model class is required (neural networks are fine). It is also the reason why many Deep RL algorithms can suffer from catastrophic forgetting.</p>
  </li>
  <li><p>When using experience replay, the TD target labels should be recalculated each time they are used.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T08:49:54.687Z
Answers.Text: Thanks. Isn't the convergence property relates to the fact that we have calculated our value function based on a bellman backup process, which is a contraction map? here, from my understanding, we use a CNN which is not really related to bellman backup
Answers.UserDisplayName: Hadar Sharvit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T08:58:49.583Z
Answers.Text: @HadarSharvit In a DQN, the CNN is being used *within* the Bellman backup process, in lieu of the Q table.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-30T08:32:15.293Z
Answers.LastActivityDate: 2022-05-30T08:57:00.427Z
Answers.LastEditDate: 2022-05-30T08:57:00.427Z
Answers.UserDisplayName: Neil Slater
Body: <p>In the original <a href="https://arxiv.org/abs/1312.5602" rel="nofollow noreferrer">DQN</a> paper, the <span class="math-container">$\ell_2$</span> loss is taken over the distance between our network output, <span class="math-container">$\hat{q}(s_j,a_j,w)$</span> and the labels <span class="math-container">$y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$</span>, where <span class="math-container">$\hat{q}(s_{j+1},a',w^-)$</span> is our network with static weights <span class="math-container">$w^-$</span>, that are updated to be <span class="math-container">$w^-=w$</span> every <span class="math-container">$C$</span> steps.</p>
  <p>This is troubling to me, as <span class="math-container">$y_j$</span> aren't really &quot;true&quot; labels as we know from supervised learning, so why should I even think that this loss updates the weights such that the output policy is something meaningful?</p>
  <p>It seems as if my network could output some arbitrary <span class="math-container">$\hat{q}$</span> and with respect to this <span class="math-container">$\hat{q}$</span>, I will try to minimize a loss, But when <span class="math-container">$\hat{q}$</span> isn't &quot;optimal&quot; per se, It is not clear that we can converge to an optimal policy.</p>
  
Tags: <reinforcement-learning><deep-rl><dqn><objective-functions>
Title: Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?
UserDisplayName: Hadar Sharvit
ThreadText: # Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?
  
  (Asked by: username_0 on 2022-05-30)
  
  In the original [DQN](https://arxiv.org/abs/1312.5602) paper, the $\ell\_2$ loss is taken over the distance between our network output, $\hat{q}(s\_j,a\_j,w)$ and the labels $y\_j=r\_j+\gamma \cdot \max\limits\_{a'} \hat{q}(s\_{j+1},a',w^-)$, where $\hat{q}(s\_{j+1},a',w^-)$ is our network with static weights $w^-$, that are updated to be $w^-=w$ every $C$ steps.
  
  This is troubling to me, as $y\_j$ aren't really "true" labels as we know from supervised learning, so why should I even think that this loss updates the weights such that the output policy is something meaningful?
  
  It seems as if my network could output some arbitrary $\hat{q}$ and with respect to this $\hat{q}$, I will try to minimize a loss, But when $\hat{q}$ isn't "optimal" per se, It is not clear that we can converge to an optimal policy.
  
  **Tags:** reinforcement-learning, deep-rl, dqn, objective-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-30. Score: 2 | Accepted answer
  
  The labels in DQN, and in Q-learning in general, are *not* "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a *current* target policy.
  
  The target policy changes every C time steps, when the network with static weights is updated. This update will include both corrections to the action value approximations, and changes to which actions are considered optimal.
  
  The reason this converges towards an optimal action value function is related to the [policy improvement theorem](http://incompleteideas.net/book/ebook/node42.html). With function approximation, as in DQN, the convergence is not guaranteed, but the process is stll based on the same idea. In summary it is a two-step repeated feedback process:
  
  1. learn value function of a current policy
  2. update the policy to select actions with maximum values
  
  What this means for the TD target "labels" in DQN is:
  
  * They are not ground truth for the optimal action value function, until *after* the algorithm has converged.
  * They are biased, initially almost completely arbitrarily by however the target network has been initialised, and from then on due to a slowly-reducing impact from that initial bias and from lagging behind collected data.
  * They are non-stationary. This means an online learning model class is required (neural networks are fine). It is also the reason why many Deep RL algorithms can suffer from catastrophic forgetting.
  * When using experience replay, the TD target labels should be recalculated each time they are used.
  
  > **Comments:**
  > **username_0**: Thanks. Isn't the convergence property relates to the fact that we have calculated our value function based on a bellman backup process, which is a contraction map? here, from my understanding, we use a CNN which is not really related to bellman backup  
  > **username_1**: @HadarSharvit In a DQN, the CNN is being used *within* the Bellman backup process, in lieu of the Q table.  
  
  ---
  
  