Answers.Body: <p>You should first read the introductory paper of Double DQN.</p>
  <p><a href="https://arxiv.org/abs/1509.06461" rel="nofollow noreferrer">https://arxiv.org/abs/1509.06461</a></p>
  <p>Then, depending on what you would like to do, search for other relevant papers that use this method.</p>
  <p>I also propose studying the original Double Q-learning paper to understand important concepts/issues, such as the overestimation bias of Q-learning: <a href="https://proceedings.neurips.cc/paper/2010/file/091d584fced301b442654dd8c23b3fc9-Paper.pdf" rel="nofollow noreferrer">https://proceedings.neurips.cc/paper/2010/file/091d584fced301b442654dd8c23b3fc9-Paper.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-28T16:27:59.663Z
Answers.LastActivityDate: 2022-11-01T16:41:35.433Z
Answers.LastEditDate: 2022-11-01T16:41:35.433Z
Answers.UserDisplayName: ddaedalus
Answers.Body: <p>If you're interested in the theory behind Double Q-learning (<strong><em>not deep!</em></strong>), the reference paper would be <a href="https://papers.nips.cc/paper/3964-double-q-learning.pdf" rel="nofollow noreferrer">Double Q-learning</a> by Hado van Hasselt (2010). </p>
  
  <p>As for Double <strong><em>deep</em></strong> Q-learning (also called DDQN, short for Double Deep Q-networks), the reference paper would be <a href="https://arxiv.org/abs/1509.06461" rel="nofollow noreferrer">Deep Reinforcement Learning with Double Q-learning</a> by Van Hasselt et al. (2016), as pointed out in
   <a href="https://ai.stackexchange.com/a/21516/34010">ddaedalus's answer</a>. </p>
  
  <p>As for how the loss is calculated, it is not explicitly written in the paper. But, you can find it in the <a href="https://arxiv.org/pdf/1511.06581.pdf" rel="nofollow noreferrer">Dueling DQN paper</a>, which is a subsequent paper where Van Hasselt is a coauthor. In the appendix, the authors provide the pseudocode for Double DQN. The relevant part for you would be:</p>
  
  <blockquote>
    <p><span class="math-container">$y_{j}=\left\{\begin{array}{ll}r &amp; \text { if } s^{\prime} \text { is terminal } \\ r+\gamma Q\left(s^{\prime}, a^{\max }\left(s^{\prime} ; \theta\right) ; \theta^{-}\right), &amp; \text {otherwise}\end{array}\right.$</span></p>
    
    <p>Do a gradient descent step with loss <span class="math-container">$ \left\|y_{j}-Q(s, a ; \theta)\right\|^{2}$</span></p>
  </blockquote>
  
  <p>Here, <span class="math-container">$y_j$</span> is the target, <span class="math-container">$\theta$</span> are the parameters of the regular network and <span class="math-container">$\theta^{-}$</span> are the target network parameters.</p>
  
  <p>The most important thing to note here is the difference with the DQN target:
  <span class="math-container">$y_{i}^{D Q N}=r+\gamma \max _{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta^{-}\right)$</span>. </p>
  
  <p>In DQN, we evaluate the Q-values based on parameters <span class="math-container">$\theta^{-}$</span> and we take the max over actions based on these Q-values parametrized with the <strong>same</strong> <span class="math-container">$\theta^{-}$</span>. The problem with this is that it leads to an overestimation bias, especially at the beginning of the training process, where the Q-values estimates are noisy. </p>
  
  <p>In order to address this issue, in double DQN, we instead take the max based on Q-values calculated using <span class="math-container">$\theta$</span> and we evaluate the Q-value of <span class="math-container">$a^{max}$</span> based on a different set of parameters i.e. <span class="math-container">$\theta^{-}$</span>.</p>
  
  <p>If you want to learn more about this, by watching a video lecture instead of reading a paper, I'd suggest you take a look at <a href="https://www.youtube.com/watch?v=7Lwf-BoIu3M&amp;t=39m20s" rel="nofollow noreferrer">this</a> lecture from UC Berkley's DRL course, where the professor (Sergey Levine) discusses this in detail with examples. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-28T18:35:07.413Z
Answers.LastActivityDate: 2020-05-28T20:09:08.107Z
Answers.LastEditDate: 2020-05-28T20:09:08.107Z
Answers.UserDisplayName: user5093249
Body: <p>I am new in reinforcement learning, but I already know deep Q-learning and Q-learning. Now, I want to learn about double deep Q-learning. </p>
  
  <p>Do you know any good references for double deep Q-learning? </p>
  
  <p>I have read some articles, but some of them don't mention what the loss is and how to calculate it, so many articles are not complete. Also, Sutton and Barto (in their book) don't describe that algorithm either. </p>
  
  <p>Please, help me to learn Double Q-learning.</p>
  
Tags: <reinforcement-learning><q-learning><reference-request><deep-rl>
Title: Is there any good reference for double deep Q-learning?
UserDisplayName: dato nefaridze
ThreadText: # Is there any good reference for double deep Q-learning?
  
  (Asked by: username_0 on 2020-05-28)
  
  I am new in reinforcement learning, but I already know deep Q-learning and Q-learning. Now, I want to learn about double deep Q-learning.
  
  Do you know any good references for double deep Q-learning?
  
  I have read some articles, but some of them don't mention what the loss is and how to calculate it, so many articles are not complete. Also, Sutton and Barto (in their book) don't describe that algorithm either.
  
  Please, help me to learn Double Q-learning.
  
  **Tags:** reinforcement-learning, q-learning, reference-request, deep-rl
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-05-28. Score: 7 | Accepted answer
  
  If you're interested in the theory behind Double Q-learning (***not deep!***), the reference paper would be [Double Q-learning](https://papers.nips.cc/paper/3964-double-q-learning.pdf) by Hado van Hasselt (2010).
  
  As for Double ***deep*** Q-learning (also called DDQN, short for Double Deep Q-networks), the reference paper would be [Deep Reinforcement Learning with Double Q-learning](https://arxiv.org/abs/1509.06461) by Van Hasselt et al. (2016), as pointed out in
  [username_1's answer](https://ai.stackexchange.com/a/21516/34010).
  
  As for how the loss is calculated, it is not explicitly written in the paper. But, you can find it in the [Dueling DQN paper](https://arxiv.org/pdf/1511.06581.pdf), which is a subsequent paper where Van Hasselt is a coauthor. In the appendix, the authors provide the pseudocode for Double DQN. The relevant part for you would be:
  
  > $y\_{j}=\left\{\begin{array}{ll}r & \text { if } s^{\prime} \text { is terminal } \\ r+\gamma Q\left(s^{\prime}, a^{\max }\left(s^{\prime} ; \theta\right) ; \theta^{-}\right), & \text {otherwise}\end{array}\right.$
  >
  > Do a gradient descent step with loss $ \left\|y\_{j}-Q(s, a ; \theta)\right\|^{2}$
  
  Here, $y\_j$ is the target, $\theta$ are the parameters of the regular network and $\theta^{-}$ are the target network parameters.
  
  The most important thing to note here is the difference with the DQN target:
  $y\_{i}^{D Q N}=r+\gamma \max \_{a^{\prime}} Q\left(s^{\prime}, a^{\prime} ; \theta^{-}\right)$.
  
  In DQN, we evaluate the Q-values based on parameters $\theta^{-}$ and we take the max over actions based on these Q-values parametrized with the **same** $\theta^{-}$. The problem with this is that it leads to an overestimation bias, especially at the beginning of the training process, where the Q-values estimates are noisy.
  
  In order to address this issue, in double DQN, we instead take the max based on Q-values calculated using $\theta$ and we evaluate the Q-value of $a^{max}$ based on a different set of parameters i.e. $\theta^{-}$.
  
  If you want to learn more about this, by watching a video lecture instead of reading a paper, I'd suggest you take a look at [this](https://www.youtube.com/watch?v=7Lwf-BoIu3M&t=39m20s) lecture from UC Berkley's DRL course, where the professor (Sergey Levine) discusses this in detail with examples.
  
  
  ---
  
  ### Answer by username_1 on 2020-05-28. Score: 5
  
  You should first read the introductory paper of Double DQN.
  
  <https://arxiv.org/abs/1509.06461>
  
  Then, depending on what you would like to do, search for other relevant papers that use this method.
  
  I also propose studying the original Double Q-learning paper to understand important concepts/issues, such as the overestimation bias of Q-learning: <https://proceedings.neurips.cc/paper/2010/file/091d584fced301b442654dd8c23b3fc9-Paper.pdf>
  
  
  ---
  
  