Answers.Body: <blockquote>
    <p>My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. </p>
  </blockquote>
  
  <p>It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You <em>could</em> cache results of the target network I suppose, but it probably would not be worth it for most environments, and I have not seen an implementation which does that.</p>
  
  <blockquote>
    <p>Am I missing something?</p>
  </blockquote>
  
  <p>It is to do with stability of the Q-learning algorithm when using function approximation (i.e. the neural network). Using a separate target network, updated every so many steps with a copy of the latest learned parameters, helps keep runaway bias from bootstrapping from dominating the system numerically, causing the estimated Q values to diverge.</p>
  
  <p>Imagine one of the data points (at <span class="math-container">$s, a, r, s'$</span>) causes a currently poor over-estimate for <span class="math-container">$q(s', a')$</span> to get worse. Maybe <span class="math-container">$s', a'$</span> has not even been visited yet, or the values of <span class="math-container">$r$</span> seen so far is higher than average, just by chance. If a sample of <span class="math-container">$(s, a)$</span> cropped up multiple times in experience replay, it would get worse again each time, because the update to <span class="math-container">$q(s,a)$</span> is based on the TD target <span class="math-container">$r + \text{max}_{a'} q(s',a')$</span>. Fixing the target network limits the damage that such over-estimates can do, giving the learning network time to converge and lose more of its initial bias.</p>
  
  <p>In this respect, using a separate target network has a very similar purpose to experience replay. It stabilises an algorithm that otherwise has problems converging.</p>
  
  <p>It is also possible to have DQN with "double learning" to address a separate issue: Maximisation bias. In that case you may see DQN implementations with 4 neural networks.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-28T19:02:07.287Z
Answers.Text: @amitection I don't understand why we need to separate two networks. Why can we not update Q in the original network? And for the problem of instability, do we use the same technique of soft updating of w for updating Q?
Answers.UserDisplayName: Prof.Plague
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-06T17:00:51.483Z
Answers.Text: For additional reading, one can refer to the paper that introduced the Double DQN: [Deep Reinforcement Learning with Double Q Learning](http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/download/12389/11847) (2016, AAAI-16) by Hado van Hasselt et al.
Answers.UserDisplayName: amitection
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-02T09:12:24.587Z
Answers.LastActivityDate: 2020-06-09T16:37:45.657Z
Answers.LastEditDate: 2020-06-09T16:37:45.657Z
Answers.UserDisplayName: Neil Slater
Body: <p>I was going through <a href="https://github.com/transedward/pytorch-dqn/blob/master/dqn_learn.py" rel="noreferrer">this</a> implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate action and the second network predicts the target Q values for finding the Bellman error.</p>
  
  <p>Why can we not just make one single network that simply predicts the Q value and use it for both the cases? My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. Is this the only reason? Am I missing something?</p>
  
Tags: <reinforcement-learning><deep-rl><q-learning><dqn><target-network>
Title: Why does DQN require two different networks?
UserDisplayName: amitection
ThreadText: # Why does DQN require two different networks?
  
  (Asked by: username_0 on 2018-07-02)
  
  I was going through [this](https://github.com/transedward/pytorch-dqn/blob/master/dqn_learn.py) implementation of DQN and I see that on line 124 and 125 two different Q networks have been initialized. From my understanding, I think one network predicts the appropriate action and the second network predicts the target Q values for finding the Bellman error.
  
  Why can we not just make one single network that simply predicts the Q value and use it for both the cases? My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. Is this the only reason? Am I missing something?
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, dqn, target-network
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-02. Score: 13 | Accepted answer
  
  > My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
  
  It has no real impact on computation time, other than a slight increase (due to extra memory used by two networks). You *could* cache results of the target network I suppose, but it probably would not be worth it for most environments, and I have not seen an implementation which does that.
  
  > Am I missing something?
  
  It is to do with stability of the Q-learning algorithm when using function approximation (i.e. the neural network). Using a separate target network, updated every so many steps with a copy of the latest learned parameters, helps keep runaway bias from bootstrapping from dominating the system numerically, causing the estimated Q values to diverge.
  
  Imagine one of the data points (at $s, a, r, s'$) causes a currently poor over-estimate for $q(s', a')$ to get worse. Maybe $s', a'$ has not even been visited yet, or the values of $r$ seen so far is higher than average, just by chance. If a sample of $(s, a)$ cropped up multiple times in experience replay, it would get worse again each time, because the update to $q(s,a)$ is based on the TD target $r + \text{max}\_{a'} q(s',a')$. Fixing the target network limits the damage that such over-estimates can do, giving the learning network time to converge and lose more of its initial bias.
  
  In this respect, using a separate target network has a very similar purpose to experience replay. It stabilises an algorithm that otherwise has problems converging.
  
  It is also possible to have DQN with "double learning" to address a separate issue: Maximisation bias. In that case you may see DQN implementations with 4 neural networks.
  
  > **Comments:**
  > **username_0**: For additional reading, one can refer to the paper that introduced the Double DQN: [Deep Reinforcement Learning with Double Q Learning](http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/download/12389/11847) (2016, AAAI-16) by Hado van Hasselt et al.  
  > **username_2**: @amitection I don't understand why we need to separate two networks. Why can we not update Q in the original network? And for the problem of instability, do we use the same technique of soft updating of w for updating Q?  
  
  ---
  
  