Answers.Body: <p>The update form <span class="math-container">$\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$</span> (where <span class="math-container">$\theta'$</span> and <span class="math-container">$\theta$</span> represent the weights of the target network and the current network, respectively) does exist and is correct. </p>
  
  <p>It is called <em>soft update</em> and it has been used in the <a href="https://arxiv.org/pdf/1509.02971.pdf" rel="nofollow noreferrer">Deep Deterministic Policy Gradient (DDPG)</a> paper, which uses the concept of a target network like DQN. The authors of the paper state that: </p>
  
  <blockquote>
    <p>The weights of these target networks are then updated by having them <strong>slowly track the learned networks</strong>: <span class="math-container">$\theta ' \leftarrow \tau \theta + (1 − \tau )\theta'$</span> with <span class="math-container">$\tau &lt;&lt; 1$</span>. This means that the target values are constrained to change slowly, greatly
    improving the stability of learning.</p>
  </blockquote>
  
  <p>This update will be made in <em>each time step</em> as follows. For example, for <span class="math-container">$\tau= 0.001$</span>, the new weights for the target network will take <span class="math-container">$0.1\%$</span> of the main network’s weights and <span class="math-container">$99.9 \%$</span> of the old target network weights. This does not go against the purpose of fixed target networks (which have been introduced to address the problem of “moving targets”). In fact, by keeping <span class="math-container">$99.9\%$</span> of the old target network weights, they can still be considered as <em>fixed</em>. </p>
  
  <p>Seeing that this resulted in improvements in DDPG, some DQN implementations/tutorials started to use soft updates for the target network. </p>
  
  <p>This is opposed to the <em>hard update</em> scheme used in the original DQN paper, i.e. the weights are copied every <span class="math-container">$C$</span> steps. This means that the target network is kept fixed for <span class="math-container">$C$</span> steps (10000 in the paper) and then gets a significant update.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-27T12:05:11.523Z
Answers.Text: okay thank you so much.So in this repository this guy updates weights after every iteration, is that correct?It works on Lunalander perfectly, like you said he used soft update and changed target network after every iteration.Is that a coincidence that algorithm works or when you use soft update you can switch networks after every iteration?
Answers.UserDisplayName: dato nefaridze
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-27T12:27:05.813Z
Answers.Text: Yes, it is correct. Instead of waiting C steps to make an update, you update your target network in each step, as follows. For example, for $\tau= 0.001$, the new weights for the target network will take $0.1\%$ of the main network’s weights and $99.9 \%$ of the old target network weights. This is not a coincidence, because recall that fixed target networks have been introduced to address the problem of “moving targets”. So, by keeping $99.9\%$ of the old target network weights, they can still be considered as fixed.
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-27T12:28:40.183Z
Answers.Text: thanks a lot, i completely understand now.
Answers.UserDisplayName: dato nefaridze
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-27T11:43:59.257Z
Answers.LastActivityDate: 2020-05-27T12:33:03.113Z
Answers.LastEditDate: 2020-05-27T12:33:03.113Z
Answers.UserDisplayName: user5093249
Body: <p>I have recently watched David silver's course, and started implementing the deep Q-learning algorithm. </p>
  
  <p>I thought I should make a switch between the Q-target and Q-current directly (meaning, every parameter of Q-current goes to Q-target), but I found a repository on GitHub where that guy updates Q-target as follows:</p>
  
  <p><span class="math-container">$$Q_{\text{target}} = \tau * Q_{\text{current}} + (1 - \tau)*Q_{\text{target}}$$</span>.</p>
  
  <p>where <span class="math-container">$\tau$</span> is some number probably between 0 and 1. </p>
  
  <p>Is that update correct or I miss something? </p>
  
  <p>I thought after some iterations (e.g. 2000 iteration), we should update Q-target as: <span class="math-container">$Q_{\text{target}}=Q_{\text{current}}$</span>.</p>
  
Tags: <deep-learning><q-learning><deep-rl><dqn>
Title: How and when should we update the Q-target in deep Q-learning?
UserDisplayName: dato nefaridze
ThreadText: # How and when should we update the Q-target in deep Q-learning?
  
  (Asked by: username_0 on 2020-05-27)
  
  I have recently watched David silver's course, and started implementing the deep Q-learning algorithm.
  
  I thought I should make a switch between the Q-target and Q-current directly (meaning, every parameter of Q-current goes to Q-target), but I found a repository on GitHub where that guy updates Q-target as follows:
  
  $$Q\_{\text{target}} = \tau \* Q\_{\text{current}} + (1 - \tau)\*Q\_{\text{target}}$$.
  
  where $\tau$ is some number probably between 0 and 1.
  
  Is that update correct or I miss something?
  
  I thought after some iterations (e.g. 2000 iteration), we should update Q-target as: $Q\_{\text{target}}=Q\_{\text{current}}$.
  
  **Tags:** deep-learning, q-learning, deep-rl, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-27. Score: 5 | Accepted answer
  
  The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct.
  
  It is called *soft update* and it has been used in the [Deep Deterministic Policy Gradient (DDPG)](https://arxiv.org/pdf/1509.02971.pdf) paper, which uses the concept of a target network like DQN. The authors of the paper state that:
  
  > The weights of these target networks are then updated by having them **slowly track the learned networks**: $\theta ' \leftarrow \tau \theta + (1 − \tau )\theta'$ with $\tau << 1$. This means that the target values are constrained to change slowly, greatly
  > improving the stability of learning.
  
  This update will be made in *each time step* as follows. For example, for $\tau= 0.001$, the new weights for the target network will take $0.1\%$ of the main network’s weights and $99.9 \%$ of the old target network weights. This does not go against the purpose of fixed target networks (which have been introduced to address the problem of “moving targets”). In fact, by keeping $99.9\%$ of the old target network weights, they can still be considered as *fixed*.
  
  Seeing that this resulted in improvements in DDPG, some DQN implementations/tutorials started to use soft updates for the target network.
  
  This is opposed to the *hard update* scheme used in the original DQN paper, i.e. the weights are copied every $C$ steps. This means that the target network is kept fixed for $C$ steps (10000 in the paper) and then gets a significant update.
  
  > **Comments:**
  > **username_0**: okay thank you so much.So in this repository this guy updates weights after every iteration, is that correct?It works on Lunalander perfectly, like you said he used soft update and changed target network after every iteration.Is that a coincidence that algorithm works or when you use soft update you can switch networks after every iteration?  
  > **username_1**: Yes, it is correct. Instead of waiting C steps to make an update, you update your target network in each step, as follows. For example, for $\tau= 0.001$, the new weights for the target network will take $0.1\%$ of the main network’s weights and $99.9 \%$ of the old target network weights. This is not a coincidence, because recall that fixed target networks have been introduced to address the problem of “moving targets”. So, by keeping $99.9\%$ of the old target network weights, they can still be considered as fixed.  
  > **username_0**: thanks a lot, i completely understand now.  
  
  ---
  
  