<p>As the authors of <a href="https://cs.brown.edu/people/kasadiat/pdf/deep_mellow.pdf" rel="nofollow noreferrer">this paper</a> state it:</p>
<blockquote>
<p>In <span class="math-container">$Q$</span>-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation.  A target network is a copy of the estimated value function that is held fixed to serve as a stable target for some number of steps.</p>
</blockquote>
<p>If I remember it correctly, the main concern is that the network could end up in a positive feedback loop, making sufficient exploration of various action and state combinations less likely to occur, which could be detrimental to the learning task.</p>

CC BY-SA 4.0
2020-07-11T11:21:30.287Z
2020-07-11T11:21:30.287Z
Daniel B.
<p>What is the idea behind double DQN?</p>
<p>The target in double DQN is computed as follows</p>
<p><span class="math-container">$$
Y_{t}^{\text {DoubleQ }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right) ; \boldsymbol{\theta}_{t}^{\prime}\right),
$$</span>
where</p>
<ul>
<li><span class="math-container">$\boldsymbol{\theta}_{t}^{\prime}$</span> are the weights of the <em>target network</em></li>
<li><span class="math-container">$\boldsymbol{\theta}_{t}$</span> are the weights of the online value network</li>
<li><span class="math-container">$\gamma$</span> is the discount factor</li>
</ul>
<p>On the other hand, the target in DQN is computed as</p>
<p><span class="math-container">$$Y_{t}^{\mathrm{DQN}} \equiv R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}^{-}\right),$$</span>
where <span class="math-container">$\boldsymbol{\theta}_{t}^{-}$</span> are the weights of the <em>target network</em>.</p>
<p>The target network for evaluating the action is updated using weights of the online network and the value fed to the target value is basically the old q value of the action.</p>
<p>Any ideas on how or why adding another network based on weights from the first network helps? Any example?</p>

<reinforcement-learning><dqn><deep-rl><double-dqn>
Why does adding another network help in double DQN?
joseph
# Why does adding another network help in double DQN?

(Asked by: username_0 on 2020-07-10)

What is the idea behind double DQN?

The target in double DQN is computed as follows

$$
Y\_{t}^{\text {DoubleQ }} \equiv R\_{t+1}+\gamma Q\left(S\_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S\_{t+1}, a ; \boldsymbol{\theta}\_{t}\right) ; \boldsymbol{\theta}\_{t}^{\prime}\right),
$$
where

* $\boldsymbol{\theta}\_{t}^{\prime}$ are the weights of the *target network*
* $\boldsymbol{\theta}\_{t}$ are the weights of the online value network
* $\gamma$ is the discount factor

On the other hand, the target in DQN is computed as

$$Y\_{t}^{\mathrm{DQN}} \equiv R\_{t+1}+\gamma \max \_{a} Q\left(S\_{t+1}, a ; \boldsymbol{\theta}\_{t}^{-}\right),$$
where $\boldsymbol{\theta}\_{t}^{-}$ are the weights of the *target network*.

The target network for evaluating the action is updated using weights of the online network and the value fed to the target value is basically the old q value of the action.

Any ideas on how or why adding another network based on weights from the first network helps? Any example?

**Tags:** reinforcement-learning, dqn, deep-rl, double-dqn


---

## 1 answers

### Answer by username_1 on 2020-07-11. Score: 1

As the authors of [this paper](https://cs.brown.edu/people/kasadiat/pdf/deep_mellow.pdf) state it:

> In $Q$-learning, the agent updates the value of executing an action in the current state, using the values of executing actions in a successive state. This procedure often results in an instability because the values change simultaneously on both sides of the update equation. A target network is a copy of the estimated value function that is held fixed to serve as a stable target for some number of steps.

If I remember it correctly, the main concern is that the network could end up in a positive feedback loop, making sufficient exploration of various action and state combinations less likely to occur, which could be detrimental to the learning task.


---

