Answers.Body: <p>Both in DQN and in DDQN, the target network starts as an exact copy of the Q-network, that has the same weights, layers, input and output dimensions, etc., as the Q-network.</p>
  <p>The main idea of the DQN agent is that the Q-network predicts the Q-values of actions from a given state and selects the maximum of them and uses the mean squared error (MSE) as its cost/loss function. That is, it performs gradient descent steps on</p>
  <p><span class="math-container">$$\left(Y_{t}^{\mathrm{DQN}} -Q\left(s_t, a_t;\boldsymbol{\theta}\right)\right)^2,$$</span></p>
  <p>where the target <span class="math-container">$Y_{t}^{\mathrm{DQN}}$</span> is defined (in the case of DQN) as</p>
  <p><span class="math-container">$$
  Y_{t}^{\mathrm{DQN}} \equiv R_{t+1}+\gamma \max _{a} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}^{-}\right)
  $$</span></p>
  <p><span class="math-container">$\boldsymbol{\theta}$</span> are the Q-network weights and <span class="math-container">$\boldsymbol{\theta^-}$</span> are the target network weights.</p>
  <p>After a usually fixed number of timesteps, the target network updates its weights by copying the weights of the Q-network. So, basically, the target network never performs a feed-forward training phase and, thus, ignores a cost function.</p>
  <p>In the case of DDQN, the target is defined as</p>
  <p><span class="math-container">$$
  Y_{t}^{\text {DDQN}} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right) ; \boldsymbol{\theta}_{t}^{-}\right)
  $$</span></p>
  <p>This target is used to decouple the selection of the action (i.e. the <em>argmax</em> part) from its evaluation (i.e. the computation of the Q value at the next state with this selected action), as stated <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">the paper that introduced the DDQN</a>)</p>
  <blockquote>
  <p>The max operator in standard Q-learning and DQN, in (2) and (3), uses the same values both to select and to evaluate an action. This makes it more likely to select overestimated values, resulting in overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-22T00:08:08.423Z
Answers.LastActivityDate: 2020-11-05T01:19:27.783Z
Answers.LastEditDate: 2020-11-05T01:19:27.783Z
Answers.UserDisplayName: ddaedalus
Body: <p>I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need to calculate the maximum Q-value for the next state <span class="math-container">$s'$</span>. In order to get that, in the DDQN case, we input that next state <span class="math-container">$s'$</span> in the target network.</p>
  <p>What I'm not clear on is: how do we train this target network itself that will help us train the other NN? What is its cost function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-21T18:42:01.797Z
Comments.Text: The loss function and method of training Double DQN are specified in the Double DQN paper: https://arxiv.org/pdf/1509.06461.pdf
Comments.UserDisplayName: rert588
Tags: <reinforcement-learning><training><dqn><deep-rl><double-dqn>
Title: How does the target network in double DQNs find the maximum Q value for each action?
UserDisplayName: Metrician
ThreadText: # How does the target network in double DQNs find the maximum Q value for each action?
  
  (Asked by: username_0 on 2020-07-21)
  
  I understand the fact that the neural network is used to take the states as inputs and it outputs the Q-value for state-action pairs. However, in order to compute this and update its weights, we need to calculate the maximum Q-value for the next state $s'$. In order to get that, in the DDQN case, we input that next state $s'$ in the target network.
  
  What I'm not clear on is: how do we train this target network itself that will help us train the other NN? What is its cost function?
  
  **Tags:** reinforcement-learning, training, dqn, deep-rl, double-dqn
  > **Comments:**
  > **username_1**: The loss function and method of training Double DQN are specified in the Double DQN paper: https://arxiv.org/pdf/1509.06461.pdf  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-22. Score: 1 | Accepted answer
  
  Both in DQN and in DDQN, the target network starts as an exact copy of the Q-network, that has the same weights, layers, input and output dimensions, etc., as the Q-network.
  
  The main idea of the DQN agent is that the Q-network predicts the Q-values of actions from a given state and selects the maximum of them and uses the mean squared error (MSE) as its cost/loss function. That is, it performs gradient descent steps on
  
  $$\left(Y\_{t}^{\mathrm{DQN}} -Q\left(s\_t, a\_t;\boldsymbol{\theta}\right)\right)^2,$$
  
  where the target $Y\_{t}^{\mathrm{DQN}}$ is defined (in the case of DQN) as
  
  $$
  Y\_{t}^{\mathrm{DQN}} \equiv R\_{t+1}+\gamma \max \_{a} Q\left(S\_{t+1}, a ; \boldsymbol{\theta}\_{t}^{-}\right)
  $$
  
  $\boldsymbol{\theta}$ are the Q-network weights and $\boldsymbol{\theta^-}$ are the target network weights.
  
  After a usually fixed number of timesteps, the target network updates its weights by copying the weights of the Q-network. So, basically, the target network never performs a feed-forward training phase and, thus, ignores a cost function.
  
  In the case of DDQN, the target is defined as
  
  $$
  Y\_{t}^{\text {DDQN}} \equiv R\_{t+1}+\gamma Q\left(S\_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S\_{t+1}, a ; \boldsymbol{\theta}\_{t}\right) ; \boldsymbol{\theta}\_{t}^{-}\right)
  $$
  
  This target is used to decouple the selection of the action (i.e. the *argmax* part) from its evaluation (i.e. the computation of the Q value at the next state with this selected action), as stated [the paper that introduced the DDQN](https://arxiv.org/pdf/1509.06461.pdf))
  
  > The max operator in standard Q-learning and DQN, in (2) and (3), uses the same values both to select and to evaluate an action. This makes it more likely to select overestimated values, resulting in overoptimistic value estimates. To prevent this, we can decouple the selection from the evaluation
  
  
  ---
  
  