Answers.Body: <p>Let's add a step index to your expression</p>
  <p><span class="math-container">$$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$</span></p>
  <p>We can expand it one step further</p>
  <p><span class="math-container">$$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$</span></p>
  <p>And further</p>
  <p><span class="math-container">$$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$</span></p>
  <p>So, I guess, we can write a general formula for <span class="math-container">$m$</span> steps behind like:</p>
  <p><span class="math-container">$$Q_{target}^{n} = (1-\tau)^{n-m}Q^{n-m}_{target} + \tau\,\sum_{i=0}^{m-1} (1-\tau)^i Q^{n-i-1}_{primary} $$</span></p>
  <p>For <span class="math-container">$n-m$</span> large enough <span class="math-container">$(1-\tau)^{n-m}$</span> should be close to 0 and you should be able to approximately reconstruct your <span class="math-container">$Q_{target}^n$</span> using only the history of <span class="math-container">$Q_{primary}$</span>es</p>
  <p><strong>Edit:</strong> I've missed that you only have snapshots with some step between them. This is not ideal, but a possible way out would be to use, say, a linear interpolation between snapshot points.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T12:35:43.350Z
Answers.Text: Sorry, I think I misunderstood previously. Did you mean if we set $m=0$ and have $n$ large enough, we will have $(1-\tau)^{n-m} = 0$ ? And also, I think you miss the fact that I only have saved weights of episodes in multiple of $n$s (say $n=10$, $Q^{10}_{primary}, Q^{20}_{primary}, ...$)? Nevertheless, your answer has very useful hints.
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T12:47:41.270Z
Answers.Text: @Sanyou Yes, sorry - was sloppy. I meant $n-m$ should be large enough. Edited the answer.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T12:50:32.153Z
Answers.Text: I see, we still need $m$, but with $n-m$ large enough the target still approximately becomes 0. With interpolation between snapshots, I think it will results in good enough approximation. I will accept your answer! Thanks a lot, excellent observation!
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T15:15:09.793Z
Answers.Text: Worth noting that although this approximation is possible, simply setting target network weights equal to primary network weights should also be OK, if the goal is to have a functional agent and interruptions are not very frequent. If your goal is to have a repeatable experiement even with interruptions, the OP would need to store a whole lot more.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T15:20:07.030Z
Answers.Text: @NeilSlater I was worried that in case when the latest primary network really spiked away from the running average it might be a bad starting point for the target network. Averaging over previous snapshots should be a more robust solution.
Answers.UserDisplayName: Kostya
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-17T05:22:01.717Z
Answers.Text: how bad is it if the approximated target is too far off from the real target? because I have used @Kostya solution with snapshot linear interpolation and the average rewards from the agent plunges and still not catching up with the previous improvement rate after 2 days of running
Answers.UserDisplayName: Sanyou
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-09T10:29:46.803Z
Answers.LastActivityDate: 2021-04-09T12:46:56.873Z
Answers.LastEditDate: 2021-04-09T12:46:56.873Z
Answers.UserDisplayName: Kostya
Body: <p>Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update strategy:</p>
  <p><span class="math-container">$$Q_{target} = (1-\tau) \times Q_{target} + \tau \times Q_{prime}$$</span></p>
  <p>I saved the primary Q network's weights every <span class="math-container">$n$</span> episodes (say <span class="math-container">$n=10$</span>), but, unfortunately, I did not save the target Q network's weights.</p>
  <p>Say that my training process is aborted for some reason, and now I would like to continue the training using the latest saved weights. I can load the primary Q network's weights, but what about the target Q network's weights? Should I also use the latest primary Q network's weights for the target Q network's weights, or should I use the primary Q network's weights from several episodes ago, or how should it be?</p>
  
Tags: <reinforcement-learning><training><deep-rl><dqn>
Title: How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?
UserDisplayName: Sanyou
ThreadText: # How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?
  
  (Asked by: username_0 on 2021-04-09)
  
  Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update strategy:
  
  $$Q\_{target} = (1-\tau) \times Q\_{target} + \tau \times Q\_{prime}$$
  
  I saved the primary Q network's weights every $n$ episodes (say $n=10$), but, unfortunately, I did not save the target Q network's weights.
  
  Say that my training process is aborted for some reason, and now I would like to continue the training using the latest saved weights. I can load the primary Q network's weights, but what about the target Q network's weights? Should I also use the latest primary Q network's weights for the target Q network's weights, or should I use the primary Q network's weights from several episodes ago, or how should it be?
  
  **Tags:** reinforcement-learning, training, deep-rl, dqn
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-09. Score: 2 | Accepted answer
  
  Let's add a step index to your expression
  
  $$Q\_{target}^{n} = (1-\tau)Q^{n-1}\_{target} + \tau\, Q^{n-1}\_{primary}$$
  
  We can expand it one step further
  
  $$Q\_{target}^{n} = (1-\tau)^2Q^{n-2}\_{target} + (1-\tau)\tau\, Q^{n-2}\_{primary} + \tau\, Q^{n-1}\_{primary}$$
  
  And further
  
  $$Q\_{target}^{n} = (1-\tau)^3Q^{n-3}\_{target} + (1-\tau)^2\tau\, Q^{n-3}\_{primary} + (1-\tau)\tau\, Q^{n-2}\_{primary} + \tau\, Q^{n-1}\_{primary}$$
  
  So, I guess, we can write a general formula for $m$ steps behind like:
  
  $$Q\_{target}^{n} = (1-\tau)^{n-m}Q^{n-m}\_{target} + \tau\,\sum\_{i=0}^{m-1} (1-\tau)^i Q^{n-i-1}\_{primary} $$
  
  For $n-m$ large enough $(1-\tau)^{n-m}$ should be close to 0 and you should be able to approximately reconstruct your $Q\_{target}^n$ using only the history of $Q\_{primary}$es
  
  **Edit:** I've missed that you only have snapshots with some step between them. This is not ideal, but a possible way out would be to use, say, a linear interpolation between snapshot points.
  
  > **Comments:**
  > **username_0**: Sorry, I think I misunderstood previously. Did you mean if we set $m=0$ and have $n$ large enough, we will have $(1-\tau)^{n-m} = 0$ ? And also, I think you miss the fact that I only have saved weights of episodes in multiple of $n$s (say $n=10$, $Q^{10}_{primary}, Q^{20}_{primary}, ...$)? Nevertheless, your answer has very useful hints.  
  > **username_1**: @username_0 Yes, sorry - was sloppy. I meant $n-m$ should be large enough. Edited the answer.  
  > **username_0**: I see, we still need $m$, but with $n-m$ large enough the target still approximately becomes 0. With interpolation between snapshots, I think it will results in good enough approximation. I will accept your answer! Thanks a lot, excellent observation!  
  > **username_2**: Worth noting that although this approximation is possible, simply setting target network weights equal to primary network weights should also be OK, if the goal is to have a functional agent and interruptions are not very frequent. If your goal is to have a repeatable experiement even with interruptions, the OP would need to store a whole lot more.  
  > **username_1**: @username_2 I was worried that in case when the latest primary network really spiked away from the running average it might be a bad starting point for the target network. Averaging over previous snapshots should be a more robust solution.  
  > **username_0**: how bad is it if the approximated target is too far off from the real target? because I have used @username_1 solution with snapshot linear interpolation and the average rewards from the agent plunges and still not catching up with the previous improvement rate after 2 days of running  
  
  ---
  
  