Answers.Body: <p>The canonical DoubleDQN uses the target network. I've not seen the first version used anywhere in the <em>deep</em> RL literature, but it looks like what one would do if they were to take the original Double Q-Learning algorithm and place it exactly into the Deep RL format. The reason they don't do this in the Double DQN paper is because it would be very expensive to train and maintain two networks, when you are using a target network anyway. I believe they found that the target network worked well enough at reducing the maximisation bias without much more computational cost to the original DQN.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-21T10:19:56.887Z
Answers.LastActivityDate: 2024-03-21T10:29:12.393Z
Answers.LastEditDate: 2024-03-21T10:29:12.393Z
Answers.UserDisplayName: David
Body: <p>I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:</p>
  <ol>
  <li>The Double DQN algorithm has two separate policies Q<sub>1</sub> and Q<sub>2</sub> with separate replay memories. They alternate in training and use the other network to get the Q values of the future state-action pair.</li>
  </ol>
  <blockquote>
  <p>Q<sub>1</sub>(s<sub>t</sub>, a) = r + γ * Q<sub>2</sub>(s<sub>t+1</sub>, argmax Q<sub>1</sub>(s<sub>t+1</sub>))</p>
  </blockquote>
  <ol start="2">
  <li>The Double DQN algorithm works like a normal DQN with a policy and a target network <em>(a copy of the policy lagging a few steps behind)</em>, and uses it to evaluate the Q value of the future state-action pair, chosen by the policy.</li>
  </ol>
  <blockquote>
  <p>Q(s<sub>t</sub>, a) = r + γ * Q<sub>target</sub>(s<sub>t+1</sub>, argmax Q(s<sub>t+1</sub>))</p>
  </blockquote>
  <p>Which one of these solutions is correct? If both of them, which one is &quot;better&quot;? Or is there a third option I haven't considered?</p>
  <p>Thanks in advance. Sorry for my math if the equations aren't correct. I hope they at least convey the intended meaning.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-21T14:44:35.830Z
Comments.Text: You can use mathjax on this site to format your formulas.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-21T14:57:35.037Z
Comments.Text: For the record I did a short experiment on DQN with two policies.  i.e. using 2 different policies with totally independent collection of history.  My results were not statistically different than DDQN with a target network.  Your results may vary.
Comments.UserDisplayName: foreverska
Tags: <reinforcement-learning><deep-rl><q-learning><dqn><double-dqn>
Title: Does the DoubleDQN algorithm use a target network or two separate policies?
UserDisplayName: Vladislav Korecký
ThreadText: # Does the DoubleDQN algorithm use a target network or two separate policies?
  
  (Asked by: username_0 on 2024-03-21)
  
  I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:
  
  1. The Double DQN algorithm has two separate policies Q1 and Q2 with separate replay memories. They alternate in training and use the other network to get the Q values of the future state-action pair.
  
  > Q1(st, a) = r + γ \* Q2(st+1, argmax Q1(st+1))
  
  2. The Double DQN algorithm works like a normal DQN with a policy and a target network *(a copy of the policy lagging a few steps behind)*, and uses it to evaluate the Q value of the future state-action pair, chosen by the policy.
  
  > Q(st, a) = r + γ \* Qtarget(st+1, argmax Q(st+1))
  
  Which one of these solutions is correct? If both of them, which one is "better"? Or is there a third option I haven't considered?
  
  Thanks in advance. Sorry for my math if the equations aren't correct. I hope they at least convey the intended meaning.
  
  **Tags:** reinforcement-learning, deep-rl, q-learning, dqn, double-dqn
  
  > **Comments:**
  > **username_1**: You can use mathjax on this site to format your formulas.  
  > **username_2**: For the record I did a short experiment on DQN with two policies.  i.e. using 2 different policies with totally independent collection of history.  My results were not statistically different than DDQN with a target network.  Your results may vary.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2024-03-21. Score: 4 | Accepted answer
  
  The canonical DoubleDQN uses the target network. I've not seen the first version used anywhere in the *deep* RL literature, but it looks like what one would do if they were to take the original Double Q-Learning algorithm and place it exactly into the Deep RL format. The reason they don't do this in the Double DQN paper is because it would be very expensive to train and maintain two networks, when you are using a target network anyway. I believe they found that the target network worked well enough at reducing the maximisation bias without much more computational cost to the original DQN.
  
  
  ---
  
  