Answers.Body: <p><span class="math-container">$$Y_{t}^{\text {DoubleDQN }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right), \boldsymbol{\theta}_{t}^{-}\right)$$</span></p>
  <p>The only difference between the &quot;original&quot; DQN and this one is that you use your <span class="math-container">$Q_\text{est}$</span> with the next state to get your action (by choosing the action with the highest Q).</p>
  <p>Afterward, you just figure out what the target <span class="math-container">$Q$</span> is given that action, by selecting the <span class="math-container">$Q$</span> belonging to that action from the target_network (instead of using the argmax a directly on the target Q network).</p>
  <p>About the formula</p>
  <ul>
  <li><p><span class="math-container">$\theta_{t}^{-}$</span> above it means frozen weights, so it represents the <em>target Q network</em>.</p>
  </li>
  <li><p>the other <span class="math-container">$\theta_{t}$</span> represents the &quot;learnable weights&quot; so the <em>estimate Q network</em>.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-16T09:17:56.203Z
Answers.Text: Yes of course, but the question is quite different, maybe I've not explained well. Try look at this question: https://stats.stackexchange.com/questions/306921/how-to-update-network-weights-in-dqn
Answers.UserDisplayName: unter_983
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-16T09:21:16.557Z
Answers.Text: In other words, "to construct a target vector" do I have to use the target network or the main network?
Answers.UserDisplayName: unter_983
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-16T11:21:21.517Z
Answers.Text: Your Q target vector that you want to converge to is the same as your output vector except for the Q at the index of the chosen action, that one is the Y_doubleDQN value described in the paper.
Answers.UserDisplayName: hal9000
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-15T23:51:14.987Z
Answers.LastActivityDate: 2020-08-17T00:13:03.097Z
Answers.LastEditDate: 2020-08-17T00:13:03.097Z
Answers.UserDisplayName: hal9000
Body: <p>I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target <span class="math-container">$y$</span> is computed, so here's how I interpreted the method (let's say I have 3 possible actions (1,2,3)):</p>
  <ol>
  <li><p>For each experience <span class="math-container">$e_{j}=(s_{j},a_{j},r_{j},s_{j+1})$</span> of the mini-batch (consider an experience where <span class="math-container">$a_{j}=2$</span>) I compute the output through the main network in the state <span class="math-container">$s_{j+1}$</span>, so I obtain 3 values.</p>
  </li>
  <li><p>I look which of the three is the highest so: <span class="math-container">$a^*=arg\max_{a}Q(s_{j+1},a)$</span>, let's say <span class="math-container">$a^*=1$</span></p>
  </li>
  <li><p>I use the target network to compute the value in <span class="math-container">$a^*=1$</span> , so  <span class="math-container">$Q_{target}(s_{j+1},1)$</span></p>
  </li>
  <li><p>I use the value at point 3 to substitute the value in the target vector associeted with the known action <span class="math-container">$a_{j}=2$</span>, so: <span class="math-container">$Q_{target}(s_{j+1},2)\leftarrow r_{j}+\gamma Q_{target}(s_{j+1},1)$</span>, while <span class="math-container">$Q_{target}(s_{j+1},1)$</span> and <span class="math-container">$Q_{target}(s_{j+1},3)$</span>, which complete the target vector <span class="math-container">$y$</span>, remain the same.</p>
  </li>
  </ol>
  <p>Is there anything wrong?</p>
  
Tags: <reinforcement-learning><q-learning><dqn><deep-rl><double-dqn>
Title: How to compute the target for double Q-learning update step?
UserDisplayName: unter_983
ThreadText: # How to compute the target for double Q-learning update step?
  
  (Asked by: username_0 on 2020-08-13)
  
  I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 possible actions (1,2,3)):
  
  1. For each experience $e\_{j}=(s\_{j},a\_{j},r\_{j},s\_{j+1})$ of the mini-batch (consider an experience where $a\_{j}=2$) I compute the output through the main network in the state $s\_{j+1}$, so I obtain 3 values.
  2. I look which of the three is the highest so: $a^\*=arg\max\_{a}Q(s\_{j+1},a)$, let's say $a^\*=1$
  3. I use the target network to compute the value in $a^\*=1$ , so $Q\_{target}(s\_{j+1},1)$
  4. I use the value at point 3 to substitute the value in the target vector associeted with the known action $a\_{j}=2$, so: $Q\_{target}(s\_{j+1},2)\leftarrow r\_{j}+\gamma Q\_{target}(s\_{j+1},1)$, while $Q\_{target}(s\_{j+1},1)$ and $Q\_{target}(s\_{j+1},3)$, which complete the target vector $y$, remain the same.
  
  Is there anything wrong?
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl, double-dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-15. Score: 1
  
  $$Y\_{t}^{\text {DoubleDQN }} \equiv R\_{t+1}+\gamma Q\left(S\_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S\_{t+1}, a ; \boldsymbol{\theta}\_{t}\right), \boldsymbol{\theta}\_{t}^{-}\right)$$
  
  The only difference between the "original" DQN and this one is that you use your $Q\_\text{est}$ with the next state to get your action (by choosing the action with the highest Q).
  
  Afterward, you just figure out what the target $Q$ is given that action, by selecting the $Q$ belonging to that action from the target\_network (instead of using the argmax a directly on the target Q network).
  
  About the formula
  
  * $\theta\_{t}^{-}$ above it means frozen weights, so it represents the *target Q network*.
  * the other $\theta\_{t}$ represents the "learnable weights" so the *estimate Q network*.
  
  > **Comments:**
  > **username_0**: Yes of course, but the question is quite different, maybe I've not explained well. Try look at this question: https://stats.stackexchange.com/questions/306921/how-to-update-network-weights-in-dqn  
  > **username_0**: In other words, "to construct a target vector" do I have to use the target network or the main network?  
  > **username_1**: Your Q target vector that you want to converge to is the same as your output vector except for the Q at the index of the chosen action, that one is the Y_doubleDQN value described in the paper.  
  
  ---
  
  