Answers.Body: <p><em>I have done some research and would like to share.</em></p>
  
  <p><strong>Generally to eliminate the use of target network one needs to show that training would be <em>stable</em> under off-policy semi-gradient.</strong> </p>
  
  <p>There are two approaches that might work:</p>
  
  <ol>
  <li>Experience reweighting</li>
  <li>Constrained optimization</li>
  </ol>
  
  <p><strong>Experience reweighting</strong></p>
  
  <p>Probably the simplest idea is to use importance sampling ratio (<a href="https://www.researchgate.net/publication/2371944" rel="nofollow noreferrer">Precup 2001</a>) to multiply each sample. This would correct the off-policy sample distribution to be on-policy distribution. It has been shown (<a href="http://jmlr.org/papers/volume17/14-488/14-488.pdf" rel="nofollow noreferrer">Sutton 2016</a>) that on-policy samples lead to stability for semi-gradient. However, this line of work that corrects for sample distribution has high variance and does not work well in practice.</p>
  
  <p>Another line of work aimed to partially correct the distribution only to the extent that would be provably stable is called <strong>Emphatic TD</strong> (<a href="http://jmlr.org/papers/volume17/14-488/14-488.pdf" rel="nofollow noreferrer">Sutton 2016</a>). The distribution is still mostly off-policy but is proved to be stable under linear function approximation.</p>
  
  <p><strong>Constrained optimization</strong></p>
  
  <p>The general wisdom comes from the fact that updating a value would also alter the target value since they share the same set of parameters. This problem is called over-generalization. To reduce this, <a href="http://www.cs.utexas.edu/~pstone/Papers/bib2html-links/NIPS17-ishand.pdf" rel="nofollow noreferrer">Durugkar</a> suggests that constraining the target Q-value after parameter update to be steady helps reduce divergence. <a href="https://arxiv.org/abs/1903.08894" rel="nofollow noreferrer">Achiam 2019</a> provides a good overview of the problem, and suggests a way to make sure that the update is non-expansion (which should prevent divergence). </p>
  
  <p><strong>Sidenote</strong></p>
  
  <p>Some works have shown convergence under off-policy samples but only in tabular case. It takes much more to show that it is stable in function approximation (even a linear one).</p>
  
  <p>Works like:</p>
  
  <ul>
  <li><strong>Q(<span class="math-container">$\lambda$</span>)</strong> (<a href="https://arxiv.org/abs/1602.04951" rel="nofollow noreferrer">Harutyunyan 2016</a>), which augments the value function with a correction term</li>
  <li><strong>Retrace(<span class="math-container">$\lambda$</span>)</strong> (<a href="http://arxiv.org/abs/1606.02647" rel="nofollow noreferrer">Munos 2016</a>), which extends the former by truncating the importance sampling ratio</li>
  </ul>
  
  <p>are shown to work <em>only</em> in tabular case. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-01T04:24:38.987Z
Answers.LastActivityDate: 2019-04-10T11:42:11.207Z
Answers.LastEditDate: 2019-04-10T11:42:11.207Z
Answers.UserDisplayName: Phizaz
Answers.Body: <p>Never tried it, but there couple of approaches which may or may not help: </p>
  
  <p>Distributional DQN ( not C51, other one, cant find ref right now typing from phone) with several output heads, chosen randomly</p>
  
  <p>Multiple agents learning randomly on each other with some regularizer to pervent collapse to same net.</p>
  
  <p>Both approach essentially try to"hide" or smear target.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-02T06:51:31.513Z
Answers.LastActivityDate: 2019-03-04T15:23:17.423Z
Answers.LastEditDate: 2019-03-04T15:23:17.423Z
Answers.UserDisplayName: mirror2image
Body: <p>In the context of Deep Q Network, a target network is usually utilized. The target network is a slow changing network with a changing rate as its hyperparameter. This includes both replacement update every <span class="math-container">$N$</span> iterations and slowly update every iteration.</p>
  
  <p>Since the rate is hard to fine tune manually, is there an alternative technique that can eliminate the use of target network or at least makes it less susceptible to the changing rate?</p>
  
Tags: <reinforcement-learning><deep-rl>
Title: Is there an alternative to the use of target network?
UserDisplayName: Phizaz
ThreadText: # Is there an alternative to the use of target network?
  
  (Asked by: username_0 on 2019-02-27)
  
  In the context of Deep Q Network, a target network is usually utilized. The target network is a slow changing network with a changing rate as its hyperparameter. This includes both replacement update every $N$ iterations and slowly update every iteration.
  
  Since the rate is hard to fine tune manually, is there an alternative technique that can eliminate the use of target network or at least makes it less susceptible to the changing rate?
  
  **Tags:** reinforcement-learning, deep-rl
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2019-03-01. Score: 6
  
  *I have done some research and would like to share.*
  
  **Generally to eliminate the use of target network one needs to show that training would be *stable* under off-policy semi-gradient.**
  
  There are two approaches that might work:
  
  1. Experience reweighting
  2. Constrained optimization
  
  **Experience reweighting**
  
  Probably the simplest idea is to use importance sampling ratio ([Precup 2001](https://www.researchgate.net/publication/2371944)) to multiply each sample. This would correct the off-policy sample distribution to be on-policy distribution. It has been shown ([Sutton 2016](http://jmlr.org/papers/volume17/14-488/14-488.pdf)) that on-policy samples lead to stability for semi-gradient. However, this line of work that corrects for sample distribution has high variance and does not work well in practice.
  
  Another line of work aimed to partially correct the distribution only to the extent that would be provably stable is called **Emphatic TD** ([Sutton 2016](http://jmlr.org/papers/volume17/14-488/14-488.pdf)). The distribution is still mostly off-policy but is proved to be stable under linear function approximation.
  
  **Constrained optimization**
  
  The general wisdom comes from the fact that updating a value would also alter the target value since they share the same set of parameters. This problem is called over-generalization. To reduce this, [Durugkar](http://www.cs.utexas.edu/~pstone/Papers/bib2html-links/NIPS17-ishand.pdf) suggests that constraining the target Q-value after parameter update to be steady helps reduce divergence. [Achiam 2019](https://arxiv.org/abs/1903.08894) provides a good overview of the problem, and suggests a way to make sure that the update is non-expansion (which should prevent divergence).
  
  **Sidenote**
  
  Some works have shown convergence under off-policy samples but only in tabular case. It takes much more to show that it is stable in function approximation (even a linear one).
  
  Works like:
  
  * **Q($\lambda$)** ([Harutyunyan 2016](https://arxiv.org/abs/1602.04951)), which augments the value function with a correction term
  * **Retrace($\lambda$)** ([Munos 2016](http://arxiv.org/abs/1606.02647)), which extends the former by truncating the importance sampling ratio
  
  are shown to work *only* in tabular case.
  
  
  ---
  
  ### Answer by username_1 on 2019-03-02. Score: 1
  
  Never tried it, but there couple of approaches which may or may not help:
  
  Distributional DQN ( not C51, other one, cant find ref right now typing from phone) with several output heads, chosen randomly
  
  Multiple agents learning randomly on each other with some regularizer to pervent collapse to same net.
  
  Both approach essentially try to"hide" or smear target.
  
  
  ---
  
  