Body: <p>I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function
  <span class="math-container">$$ L_i(\theta_i) = [(y_i - Q(s,a;\theta_i))^2] $$</span>
  where
  <span class="math-container">$$ y_i = r + \gamma * max_{a'}Q(s',a';\theta_{i-1}) $$</span>
  is the target value in the loss function.</p>
  
  <p>From my understanding, experiences are pulled from the replay buffer, then the DQN is used to estimate the future sum of the discounted rewards for the next state (assuming it plays optimally) and adds this onto the current rewards <span class="math-container">$r$</span> to create the target value. Then the DQN is used again to estimate <span class="math-container">$Q$</span> value for the current state. Then the loss function is just the difference between the target and the estimated <span class="math-container">$Q$</span> value for the current state. Afterward, you optimize the loss function.</p>
  
  <p>But, if the parameters of the DQN start off randomly, then surely the target value will be completely wrong since the parameters that define that target function are random. So, if the target function is wrong, then it will minimize the difference between the target value and the estimated value, but it will be learning to predict incorrect values, since the target values are wrong?</p>
  
  <p>I don't understand why the target value works if the parameters of the DQN needed to create that target are completely random.</p>
  
  <p>What obvious mistake am I making here?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-02T18:44:19.030Z
Comments.Text: Normally when calculating the target $y_i$, we should use parameters/weights $\theta^{-}$ of the target network, which are a periodically-updated copy of the main network parameters $\theta$ (See [here](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf "here")). So, just checking if this is a typo in your notation, and if even when fixed, your doubts still exist.
Comments.UserDisplayName: user5093249
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-04T14:43:56.360Z
Comments.Text: Note that, in the case of tabular Q-learning (Q-learning without function approximation), the initial targets are also noisy, but hopefully they will converge to the correct target, as you update the state-action value function and explore (and exploit) the environment.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-04T16:22:24.250Z
Comments.Text: Thank you for the replies. I think I have a better idea about what is happening.
Comments.UserDisplayName: Alex Obrien
Tags: <reinforcement-learning><q-learning><deep-rl><dqn><objective-functions>
Title: How can the target rely on untrained parameters?
UserDisplayName: Alex Obrien
ThreadText: # How can the target rely on untrained parameters?
  
  (Asked by: username_0 on 2020-06-02)
  
  I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function
  $$ L\_i(\theta\_i) = [(y\_i - Q(s,a;\theta\_i))^2] $$
  where
  $$ y\_i = r + \gamma \* max\_{a'}Q(s',a';\theta\_{i-1}) $$
  is the target value in the loss function.
  
  From my understanding, experiences are pulled from the replay buffer, then the DQN is used to estimate the future sum of the discounted rewards for the next state (assuming it plays optimally) and adds this onto the current rewards $r$ to create the target value. Then the DQN is used again to estimate $Q$ value for the current state. Then the loss function is just the difference between the target and the estimated $Q$ value for the current state. Afterward, you optimize the loss function.
  
  But, if the parameters of the DQN start off randomly, then surely the target value will be completely wrong since the parameters that define that target function are random. So, if the target function is wrong, then it will minimize the difference between the target value and the estimated value, but it will be learning to predict incorrect values, since the target values are wrong?
  
  I don't understand why the target value works if the parameters of the DQN needed to create that target are completely random.
  
  What obvious mistake am I making here?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, dqn, objective-functions
  > **Comments:**
  > **username_1**: Normally when calculating the target $y_i$, we should use parameters/weights $\theta^{-}$ of the target network, which are a periodically-updated copy of the main network parameters $\theta$ (See [here](https://web.stanford.edu/class/psych209/Readings/MnihEtAlHassibis15NatureControlDeepRL.pdf "here")). So, just checking if this is a typo in your notation, and if even when fixed, your doubts still exist.  
  > **username_2**: Note that, in the case of tabular Q-learning (Q-learning without function approximation), the initial targets are also noisy, but hopefully they will converge to the correct target, as you update the state-action value function and explore (and exploit) the environment.  
  > **username_0**: Thank you for the replies. I think I have a better idea about what is happening.  
  
  ---
  
  ## 0 answers
  
  