Answers.Body: <blockquote>
    <p>However, if you use a target network, which is less trained than the normal net — especially at early stages of the training — wouldn't the state/action/advantage value be updating towards an inferior estimate?</p>
  </blockquote>
  
  <p>Possibly, but a critical part of <em>stability</em> of TD learning with function approximation when adding a target network is that any updates will be consistent in the short term.</p>
  
  <p>Consider that with a single network, the TD target calculation will also be biased (and by the same amounts on the first few passes), but they will not be consistent. Each learning update shifts the estimates by a value that is biased. Problems occur with stability when instead of the bias decaying over time due to real data from each step, that the bias amount is enough to form a positive feedback loop. </p>
  
  <p>As a concrete example, suppose an initial random network calculating <span class="math-container">$\hat{q}(s,a,\theta)$</span> where</p>
  
  <ul>
  <li>The reward is 1.0</li>
  <li>The true value function <span class="math-container">$q^*(s,a)$</span> is 10.0 (it is not really relevant) and that would make <span class="math-container">$q^*(s',a') = 9.0$</span></li>
  <li><p>The NN initially, predicts 5.0 for <span class="math-container">$\hat{q}(s,a,\theta)$</span> and 7.0 for <span class="math-container">$\text{max}_{a'}\hat{q}(s',a',\theta)$</span>. </p></li>
  <li><p>We also have a learning network using parameters <span class="math-container">$\theta$</span> that change on each learning update. </p></li>
  <li><p>We have a frozen copy of the initial network using parameter <span class="math-container">$\bar{\theta}$</span> </p></li>
  <li><p>We have some learning rate <span class="math-container">$\alpha$</span> that for arguments sake we say reduces error in this case by 50% each time it sees this example.</p></li>
  <li><p>The approximation process in the NN means that <span class="math-container">$\hat{q}(s,a,\theta)$</span> and <span class="math-container">$\text{max}_{a'}\hat{q}(s',a',\theta)$</span> are linked. This is a real feature of neural networks, but hard to model as it evolves over time. Let's say that with the initial setup that any learning of <span class="math-container">$\hat{q}(s,a,\theta)$</span> in isolation will make <span class="math-container">$\text{max}_{a'}\hat{q}(s',a',\theta)$</span> change towards <span class="math-container">$\hat{q}(s,a,\theta)$</span> by a 50% step. These steps sizes are not that important, they could be just 1% and the problem still occurs.</p>
  
  <ul>
  <li>You can control the learning rate directly, but cannot really control the "generalisation link strength" of the approximation.</li>
  </ul></li>
  <li><p>To really trigger the problem I want to show, let's say that these single steps are occurring far enough away from the end of an episode, or other moderating feedback, that we effectively get a few hundred visits to our <span class="math-container">$(s,a)$</span> pair without impact from other items (in fact it is likely that many will be going through the same issues)</p></li>
  </ul>
  
  <p>Without the target network, the first TD target is <span class="math-container">$1.0 + 7.0 = 8.0$</span>. After update <span class="math-container">$\hat{q}(s,a,\theta) = 6.5$</span> and <span class="math-container">$\text{max}_{a'}\hat{q}(s',a',\theta)$</span> = 6.75$. This looks good, right? Getting closer to the real values . . . but keep going . . . the next updates work like this:</p>
  
  <pre><code>q(s,a)      max q(s',a')
   7.125       6.938
   7.531       7.234
   7.882       7.559
   8.221       7.889
  </code></pre>
  
  <p>Again, this looks OK? But let's come back to it after 100 [isolated, so a bit fake] iterations:</p>
  
  <pre><code>40.222      39.889
  40.556      40.222
  40.889      40.556
  41.222      40.889
  41.556      41.222
  </code></pre>
  
  <p>This has overshot, and both values are increasing exponentially. The initial neural network bias is caught in a positive feedback loop.</p>
  
  <p>Now if we use the target network that always predicts 7.0, what happens is what you expect. After 100 iterations we have:</p>
  
  <pre><code> 8.0         7.5
   8.0         7.5
   8.0         7.5
   8.0         7.5
   8.0         7.5
  </code></pre>
  
  <p>These values are still incorrect, but the updates have made a more conservative and stable step towards the correct values. Note the second value is what the learning network would <em>predict</em> for <span class="math-container">$\text{max}_{a'}\hat{q}(s',a',\theta)$</span>, but we have used the other prediction <span class="math-container">$\text{max}_{a'}\hat{q}(s',a',\bar{\theta})$</span> on each step.</p>
  
  <p>In reality the feedback loops are more complex that this, because this is function approximation so action value estimates from different <span class="math-container">$(s,a)$</span> pairs interact in ways that are hard to predict. But the worry is that bias causes divergence in estimates, and this does happen in practice. It is more likely to happen in environments with long episodes, or where state loops are possible.</p>
  
  <p>It is also worth noting that using the frozen target network has not <em>solved</em> the problem, it has just significantly throttled runaway feedback. The amount of throttling required to keep learning stable will vary depending on the problem. The number of steps between target network updates is a hyperparameter. Set it too low and you risk seeing stability problems. Set it too high and learning will take longer.</p>
  
  <blockquote>
    <p>I've tried implementing DQNs and DDPGs on Cartpole, and I've found that the algorithms fail to converge when target networks are used, but work fine when those target networks are removed.</p>
  </blockquote>
  
  <p>In that case, your implementations are incorrect. Definitely I have observed DQN with a target network working many times on Cartpole.</p>
  
  <p>Whether or not using a target network makes convergence faster or more stable is more complex, and it may be for your network design or hyperparameter choices that adding the target network is making performance worse.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-20T08:35:36.493Z
Answers.LastActivityDate: 2019-07-20T08:35:36.493Z
Answers.UserDisplayName: Neil Slater
Body: <p>I'm having some trouble understanding some parts of the usage of target networks. </p>
  
  <p>I get that having the same network predict the state/action/advantage values for both the current networks can lead to instability. </p>
  
  <p>Based on my understanding, the intuition behind 1-step TD error that going a step into the future will give you a better estimate, that can then be used to update your original state/action/advantage value.</p>
  
  <p>However, if you use a target network, which is less trained than the normal net — especially at early stages of the training — wouldn't the state/action/advantage value be updating towards an inferior estimate?</p>
  
  <p>I've tried implementing DQNs and DDPGs on Cartpole, and I've found that the algorithms fail to converge when target networks are used, but work fine when those target networks are removed. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-20T14:29:00.527Z
Comments.Text: DQN convergence is so slow (and often non-monotonous) that difference in prediction between target and training networks is not noticeable.
Comments.UserDisplayName: mirror2image
Tags: <reinforcement-learning><q-learning><dqn><deep-rl>
Title: Will the target network, which is less trained than the normal network, output inferior estimates?
UserDisplayName: Alex
ThreadText: # Will the target network, which is less trained than the normal network, output inferior estimates?
  
  (Asked by: username_0 on 2019-07-20)
  
  I'm having some trouble understanding some parts of the usage of target networks.
  
  I get that having the same network predict the state/action/advantage values for both the current networks can lead to instability.
  
  Based on my understanding, the intuition behind 1-step TD error that going a step into the future will give you a better estimate, that can then be used to update your original state/action/advantage value.
  
  However, if you use a target network, which is less trained than the normal net — especially at early stages of the training — wouldn't the state/action/advantage value be updating towards an inferior estimate?
  
  I've tried implementing DQNs and DDPGs on Cartpole, and I've found that the algorithms fail to converge when target networks are used, but work fine when those target networks are removed.
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl
  
  > **Comments:**
  > **username_1**: DQN convergence is so slow (and often non-monotonous) that difference in prediction between target and training networks is not noticeable.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-07-20. Score: 2 | Accepted answer
  
  > However, if you use a target network, which is less trained than the normal net — especially at early stages of the training — wouldn't the state/action/advantage value be updating towards an inferior estimate?
  
  Possibly, but a critical part of *stability* of TD learning with function approximation when adding a target network is that any updates will be consistent in the short term.
  
  Consider that with a single network, the TD target calculation will also be biased (and by the same amounts on the first few passes), but they will not be consistent. Each learning update shifts the estimates by a value that is biased. Problems occur with stability when instead of the bias decaying over time due to real data from each step, that the bias amount is enough to form a positive feedback loop.
  
  As a concrete example, suppose an initial random network calculating $\hat{q}(s,a,\theta)$ where
  
  * The reward is 1.0
  * The true value function $q^\*(s,a)$ is 10.0 (it is not really relevant) and that would make $q^\*(s',a') = 9.0$
  * The NN initially, predicts 5.0 for $\hat{q}(s,a,\theta)$ and 7.0 for $\text{max}\_{a'}\hat{q}(s',a',\theta)$.
  * We also have a learning network using parameters $\theta$ that change on each learning update.
  * We have a frozen copy of the initial network using parameter $\bar{\theta}$
  * We have some learning rate $\alpha$ that for arguments sake we say reduces error in this case by 50% each time it sees this example.
  * The approximation process in the NN means that $\hat{q}(s,a,\theta)$ and $\text{max}\_{a'}\hat{q}(s',a',\theta)$ are linked. This is a real feature of neural networks, but hard to model as it evolves over time. Let's say that with the initial setup that any learning of $\hat{q}(s,a,\theta)$ in isolation will make $\text{max}\_{a'}\hat{q}(s',a',\theta)$ change towards $\hat{q}(s,a,\theta)$ by a 50% step. These steps sizes are not that important, they could be just 1% and the problem still occurs.
  
    + You can control the learning rate directly, but cannot really control the "generalisation link strength" of the approximation.
  * To really trigger the problem I want to show, let's say that these single steps are occurring far enough away from the end of an episode, or other moderating feedback, that we effectively get a few hundred visits to our $(s,a)$ pair without impact from other items (in fact it is likely that many will be going through the same issues)
  
  Without the target network, the first TD target is $1.0 + 7.0 = 8.0$. After update $\hat{q}(s,a,\theta) = 6.5$ and $\text{max}\_{a'}\hat{q}(s',a',\theta)$ = 6.75$. This looks good, right? Getting closer to the real values . . . but keep going . . . the next updates work like this:
  
  ```
  q(s,a)      max q(s',a')
   7.125       6.938
   7.531       7.234
   7.882       7.559
   8.221       7.889
  
  ```
  
  Again, this looks OK? But let's come back to it after 100 [isolated, so a bit fake] iterations:
  
  ```
  40.222      39.889
  40.556      40.222
  40.889      40.556
  41.222      40.889
  41.556      41.222
  
  ```
  
  This has overshot, and both values are increasing exponentially. The initial neural network bias is caught in a positive feedback loop.
  
  Now if we use the target network that always predicts 7.0, what happens is what you expect. After 100 iterations we have:
  
  ```
   8.0         7.5
   8.0         7.5
   8.0         7.5
   8.0         7.5
   8.0         7.5
  
  ```
  
  These values are still incorrect, but the updates have made a more conservative and stable step towards the correct values. Note the second value is what the learning network would *predict* for $\text{max}\_{a'}\hat{q}(s',a',\theta)$, but we have used the other prediction $\text{max}\_{a'}\hat{q}(s',a',\bar{\theta})$ on each step.
  
  In reality the feedback loops are more complex that this, because this is function approximation so action value estimates from different $(s,a)$ pairs interact in ways that are hard to predict. But the worry is that bias causes divergence in estimates, and this does happen in practice. It is more likely to happen in environments with long episodes, or where state loops are possible.
  
  It is also worth noting that using the frozen target network has not *solved* the problem, it has just significantly throttled runaway feedback. The amount of throttling required to keep learning stable will vary depending on the problem. The number of steps between target network updates is a hyperparameter. Set it too low and you risk seeing stability problems. Set it too high and learning will take longer.
  
  > I've tried implementing DQNs and DDPGs on Cartpole, and I've found that the algorithms fail to converge when target networks are used, but work fine when those target networks are removed.
  
  In that case, your implementations are incorrect. Definitely I have observed DQN with a target network working many times on Cartpole.
  
  Whether or not using a target network makes convergence faster or more stable is more complex, and it may be for your network design or hyperparameter choices that adding the target network is making performance worse.
  
  
  ---
  
  