Answers.Body: <p>I think I'm not able to fully answer this, although I try to provide some insights:</p>
  <ul>
  <li>In regular DQN with target network, the training objective of the Q-network is to estimate the Q-value of the current state-action pair: <span class="math-container">$Q(s_t, a_t;\theta)\approx r_t + \gamma \max_{a_{t+1}}Q(s_{t+1}, a_{t+1}; \theta^-)$</span>, where <span class="math-container">$\theta$</span> are the online parameters and <span class="math-container">$\theta^-$</span> the target parameters.</li>
  <li>Now if you &quot;reverse&quot; the target network, the quantity <span class="math-container">$\max_{a_{t+1}}Q(s_{t+1}, a_{t+1}; \theta)$</span> is not anymore fixed but instead it <em>moves</em> at each update of <span class="math-container">$\theta$</span> (which is usually at each timestep.) An instead you held fixed the current action-state value: <span class="math-container">$Q(s_t,a_t;\theta^-)$</span>.</li>
  </ul>
  <p>I guess this could make training more unstable because the regression target is not anymore fixed but moving: I guess this can be more problematic when <span class="math-container">$Q(s_{t+1}, a_{t+1})$</span> is quite different than <span class="math-container">$Q(s_t, a_t)$</span>; although the inverse target may still be good at decorrelating.</p>
  <p>Indeed, this should be checked on challenging environments, like Atari, in order to have a clear picture on this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T10:04:55.637Z
Answers.LastActivityDate: 2023-04-26T10:04:55.637Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am not asking why using a Target Network is useful (this was very well explained <a href="https://ai.stackexchange.com/questions/22504/why-do-we-need-target-network-in-deep-q-learning">here</a>), but rather if using this &quot;inversed&quot; target network is equivalent:</p>
  <p><span class="math-container">$\left(r_t + \max_aQ(s_{t+1},a;\theta) - Q(s_t,a_t; \theta^-)\right)^2$</span> were <span class="math-container">$\theta^-$</span> is some old version of the parameters that gets updated every <span class="math-container">$C \in \mathbb{N}$</span> updates, and the Q-Network with these parameters is the target network.</p>
  <p>Basically I'm only updating the one used to determine the expected q value at next iteration instead of only updating the one used to determine the best action. And the one I'm not updating gets updated each <span class="math-container">$C$</span> iteration.</p>
  <p>Both seem to converge in my MREs but probably because the environment I'm in is &quot;too simple&quot; to see the benefits of a Target Network.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-25T19:58:51.547Z
Comments.Text: So you're updating $\theta$ at each iteration, and $\theta^-$ every $C$ iterations, right?
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-04-25T21:06:43.430Z
Comments.Text: Correct, this is what I'm doing.
Comments.UserDisplayName: FluidMechanics Potential Flows
Tags: <reinforcement-learning><deep-learning>
Title: Target Network inversed in Deep Q Learning (Reinforcement Learning)
UserDisplayName: FluidMechanics Potential Flows
ThreadText: # Target Network inversed in Deep Q Learning (Reinforcement Learning)
  
  (Asked by: username_0 on 2023-04-24)
  
  I am not asking why using a Target Network is useful (this was very well explained [here](https://ai.stackexchange.com/questions/22504/why-do-we-need-target-network-in-deep-q-learning)), but rather if using this "inversed" target network is equivalent:
  
  $\left(r\_t + \max\_aQ(s\_{t+1},a;\theta) - Q(s\_t,a\_t; \theta^-)\right)^2$ were $\theta^-$ is some old version of the parameters that gets updated every $C \in \mathbb{N}$ updates, and the Q-Network with these parameters is the target network.
  
  Basically I'm only updating the one used to determine the expected q value at next iteration instead of only updating the one used to determine the best action. And the one I'm not updating gets updated each $C$ iteration.
  
  Both seem to converge in my MREs but probably because the environment I'm in is "too simple" to see the benefits of a Target Network.
  
  **Tags:** reinforcement-learning, deep-learning
  
  > **Comments:**
  > **username_1**: So you're updating $\theta$ at each iteration, and $\theta^-$ every $C$ iterations, right?  
  > **username_0**: Correct, this is what I'm doing.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-26. Score: 2
  
  I think I'm not able to fully answer this, although I try to provide some insights:
  
  * In regular DQN with target network, the training objective of the Q-network is to estimate the Q-value of the current state-action pair: $Q(s\_t, a\_t;\theta)\approx r\_t + \gamma \max\_{a\_{t+1}}Q(s\_{t+1}, a\_{t+1}; \theta^-)$, where $\theta$ are the online parameters and $\theta^-$ the target parameters.
  * Now if you "reverse" the target network, the quantity $\max\_{a\_{t+1}}Q(s\_{t+1}, a\_{t+1}; \theta)$ is not anymore fixed but instead it *moves* at each update of $\theta$ (which is usually at each timestep.) An instead you held fixed the current action-state value: $Q(s\_t,a\_t;\theta^-)$.
  
  I guess this could make training more unstable because the regression target is not anymore fixed but moving: I guess this can be more problematic when $Q(s\_{t+1}, a\_{t+1})$ is quite different than $Q(s\_t, a\_t)$; although the inverse target may still be good at decorrelating.
  
  Indeed, this should be checked on challenging environments, like Atari, in order to have a clear picture on this.
  
  
  ---
  
  