Answers.Body: <p>In DQN that was presented in the <a href="https://www.nature.com/articles/nature14236" rel="noreferrer">original paper</a> the update target for the Q-Network is <span class="math-container">$\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$</span> were <span class="math-container">$\theta^-$</span> is some old version of the parameters that gets updated every <span class="math-container">$C$</span> updates, and the Q-Network with these parameters is the target network.</p>
  <p>If you didn't use this target network, i.e. if your update target was <span class="math-container">$\left(r_t + \max_aQ(s_{t+1},a;\theta) - Q(s_t,a_t; \theta)\right)^2$</span>,  then learning would become unstable because the target, <span class="math-container">$r_t + \max_aQ(s_{t+1},a;\theta)$</span>, and the prediction, <span class="math-container">$Q(s_t,a_t; \theta)$</span>, are not independent, as they both rely on <span class="math-container">$\theta$</span>.</p>
  <p>A nice analogy I saw once was that it is akin to a dog chasing it's own tail - it will never catch it because the target is non-stationary; this non-stationarity is exactly what the dependence between the target and the prediction cause.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T03:16:27.090Z
Answers.Text: Given that this is a duplicate but your answer is valuable, I think it would have been better to write the answer under https://ai.stackexchange.com/q/6982/2444.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-15T21:53:38.770Z
Answers.LastActivityDate: 2020-07-15T22:33:13.610Z
Answers.LastEditDate: 2020-07-15T22:33:13.610Z
Answers.UserDisplayName: David
Body: <p>I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep Q-learning deeply, because I am about to make some changes in the deep Q-learning algorithm (i.e. invent a new one). Can you help me to understand what happens during executing a deep Q-learning algorithm intuitively?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-15T20:54:53.390Z
Comments.Text: I think also this question goes into the exact same direction: [Why does adding another network help in double DQN?](https://ai.stackexchange.com/questions/22443/why-does-adding-another-network-help-in-double-dqn).
Comments.UserDisplayName: Daniel B.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-15T21:01:15.943Z
Comments.Text: By the way: Here is a video that also explains it nicely as well: https://youtu.be/xVkPh9E9GfE?t=171
  I can only recommend that online lecture series.
Comments.UserDisplayName: Daniel B.
Tags: <deep-learning><q-learning><dqn><deep-rl><deep-neural-networks>
Title: Why do we need target network in deep Q learning?
UserDisplayName: dato nefaridze
ThreadText: # Why do we need target network in deep Q learning?
  
  (Asked by: username_0 on 2020-07-15)
  
  I already know deep RL, but to learn it deeply I want to know why do we need 2 networks in deep RL. What does the target network do? I now there is huge mathematics into this, but I want to know deep Q-learning deeply, because I am about to make some changes in the deep Q-learning algorithm (i.e. invent a new one). Can you help me to understand what happens during executing a deep Q-learning algorithm intuitively?
  
  **Tags:** deep-learning, q-learning, dqn, deep-rl, deep-neural-networks
  > **Comments:**
  > **username_1**: I think also this question goes into the exact same direction: [Why does adding another network help in double DQN?](https://ai.stackexchange.com/questions/22443/why-does-adding-another-network-help-in-double-dqn).  
  > **username_1**: By the way: Here is a video that also explains it nicely as well: https://youtu.be/xVkPh9E9GfE?t=171 I can only recommend that online lecture series.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-15. Score: 7
  
  In DQN that was presented in the [original paper](https://www.nature.com/articles/nature14236) the update target for the Q-Network is $\left(r\_t + \max\_aQ(s\_{t+1},a;\theta^-) - Q(s\_t,a\_t; \theta)\right)^2$ were $\theta^-$ is some old version of the parameters that gets updated every $C$ updates, and the Q-Network with these parameters is the target network.
  
  If you didn't use this target network, i.e. if your update target was $\left(r\_t + \max\_aQ(s\_{t+1},a;\theta) - Q(s\_t,a\_t; \theta)\right)^2$, then learning would become unstable because the target, $r\_t + \max\_aQ(s\_{t+1},a;\theta)$, and the prediction, $Q(s\_t,a\_t; \theta)$, are not independent, as they both rely on $\theta$.
  
  A nice analogy I saw once was that it is akin to a dog chasing it's own tail - it will never catch it because the target is non-stationary; this non-stationarity is exactly what the dependence between the target and the prediction cause.
  
  > **Comments:**
  > **username_3**: Given that this is a duplicate but your answer is valuable, I think it would have been better to write the answer under https://ai.stackexchange.com/q/6982/2444.  
  
  ---
  
  