Answers.Body: <p>As the authors of <a href="https://cs.brown.edu/people/kasadiat/pdf/deep_mellow.pdf" rel="nofollow noreferrer">this paper</a> state it:</p>
  <blockquote>
  <p>In <span class="math-container">$Q$</span>-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation.  A target network is a copy of the estimated value function that is held fixed to serve as a stable target for some number of steps.</p>
  </blockquote>
  <p>If I remember it correctly, the main concern is that the network could end up in a positive feedback loop, making sufficient exploration of various action and state combinations less likely to occur, which could be detrimental to the learning task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-11T11:21:30.287Z
Answers.LastActivityDate: 2020-07-11T11:21:30.287Z
Answers.UserDisplayName: Daniel B.
Body: <p>What is the idea behind double DQN?</p>
  <p>The target in double DQN is computed as follows</p>
  <p><span class="math-container">$$
  Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right) ; \boldsymbol{\theta}_{t}^{\prime}\right),
  $$</span>
  where</p>
  <ul>
  <li><span class="math-container">$\boldsymbol{\theta}_{t}^{\prime}$</span> are the weights of the <em>target network</em></li>
  <li><span class="math-container">$\boldsymbol{\theta}_{t}$</span> are the weights of the online value network</li>
  <li><span class="math-container">$\gamma$</span> is the discount factor</li>
  </ul>
  <p>On the other hand, the target in DQN is computed as</p>
  <p><span class="math-container">$$Y_{t}^{\mathrm{DQN}} \equiv R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}^{-}\right),$$</span>
  where <span class="math-container">$\boldsymbol{\theta}_{t}^{-}$</span> are the weights of the <em>target network</em>.</p>
  <p>The target network for evaluating the action is updated using weights of the online network and the value fed to the target value is basically the old q value of the action.</p>
  <p>Any ideas on how or why adding another network based on weights from the first network helps? Any example?</p>
  
Tags: <reinforcement-learning><dqn><deep-rl><double-dqn>
Title: Why does adding another network help in double DQN?
UserDisplayName: joseph
ThreadText: # Why does adding another network help in double DQN?
  
  (Asked by: username_0 on 2020-07-10)
  
  What is the idea behind double DQN?
  
  The target in double DQN is computed as follows
  
  $$
  Y\_{t}^{\text {DoubleQ }} \equiv R\_{t+1}+\gamma Q\left(S\_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S\_{t+1}, a ; \boldsymbol{\theta}\_{t}\right) ; \boldsymbol{\theta}\_{t}^{\prime}\right),
  $$
  where
  
  * $\boldsymbol{\theta}\_{t}^{\prime}$ are the weights of the *target network*
  * $\boldsymbol{\theta}\_{t}$ are the weights of the online value network
  * $\gamma$ is the discount factor
  
  On the other hand, the target in DQN is computed as
  
  $$Y\_{t}^{\mathrm{DQN}} \equiv R\_{t+1}+\gamma \max \_{a} Q\left(S\_{t+1}, a ; \boldsymbol{\theta}\_{t}^{-}\right),$$
  where $\boldsymbol{\theta}\_{t}^{-}$ are the weights of the *target network*.
  
  The target network for evaluating the action is updated using weights of the online network and the value fed to the target value is basically the old q value of the action.
  
  Any ideas on how or why adding another network based on weights from the first network helps? Any example?
  
  **Tags:** reinforcement-learning, dqn, deep-rl, double-dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-11. Score: 1
  
  As the authors of [this paper](https://cs.brown.edu/people/kasadiat/pdf/deep_mellow.pdf) state it:
  
  > In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation. A target network is a copy of the estimated value function that is held fixed to serve as a stable target for some number of steps.
  
  If I remember it correctly, the main concern is that the network could end up in a positive feedback loop, making sufficient exploration of various action and state combinations less likely to occur, which could be detrimental to the learning task.
  
  
  ---
  
  