Answers.Body: <p>There is no thorough proof, theoretical or experimental that Double DQN is better then vanilla DQN. There are a lot of different tasks, paper and later experiments only explore some of them. What practitioner can take out of it is that <em>on some tasks</em> DDQN is better. That's the essence of Deep Mind's "Rainbow" approach - drop a lot of different methods into bucket and take best results. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T12:13:21.223Z
Answers.Text: Can you give some references where DQN outperforms DoubleDQN?
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T12:14:34.747Z
Answers.Text: I have a problem where my DQN is better than DoubleDQN and I cannot give a valid explanation on why.
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T07:35:29.923Z
Answers.LastActivityDate: 2019-04-09T07:35:29.923Z
Answers.UserDisplayName: mirror2image
Answers.Body: <p>That may happen when the value of the state is bad. You can find the example and explain about that in the link below.</p>
  
  <p>See this:<a href="https://medium.freecodecamp.org/improvements-in-deep-q-learning-dueling-double-dqn-prioritized-experience-replay-and-fixed-58b130cc5682" rel="nofollow noreferrer">https://medium.freecodecamp.org/improvements-in-deep-q-learning-dueling-double-dqn-prioritized-experience-replay-and-fixed-58b130cc5682</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T07:53:45.827Z
Answers.Text: What do you mean by "the value of the state is bad"? As I'm using an OpenAI-Gym environment, the value of the state is just the observation that I'm getting from it.
Answers.UserDisplayName: Angelo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T08:11:27.903Z
Answers.Text: @Angelo  you can read the blog above to understand the answer . As you calculate all actions at one state but all these actions  do not affect the environment in a relevant way.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T08:37:43.857Z
Answers.Text: @Angelo In most states, the choice of action has no effect.
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-04T21:23:03.623Z
Answers.Text: We prefer answers that contain the necessary info to answer the question, rather than answers that just link to another external resource, so I suggest that you provide the necessary info from that article in your answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T12:16:41.687Z
Answers.Text: In your link, it says that in DuelingDQN session, not in DoubleDQN as asked.
Answers.UserDisplayName: ddaedalus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T14:57:22.167Z
Answers.Text: @ddaedalus It has both of them and you can see this: https://medium.com/@parsa_h_m/deep-reinforcement-learning-dqn-double-dqn-dueling-dqn-noisy-dqn-and-dqn-with-prioritized-551f621a9823
Answers.UserDisplayName: I_Al-thamary
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-09T04:00:04.243Z
Answers.LastActivityDate: 2019-04-09T04:00:04.243Z
Answers.UserDisplayName: I_Al-thamary
Body: <p>I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen <a href="https://arxiv.org/pdf/1509.06461.pdf" rel="nofollow noreferrer">in the double DQN paper</a>, double DQN should perform better than DQN. Am I doing something wrong or is it possible?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-10T12:18:20.703Z
Comments.Text: Hello. I am facing the same issue. What explanation finally did you give ?
Comments.UserDisplayName: ddaedalus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T16:28:34.750Z
Comments.Text: I have the same result, DQN performs better than Double DQN. I m try to understand why.
Comments.UserDisplayName: Mustafa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-10T13:26:39.307Z
Comments.Text: DDQN was designed to address Q-function overestimation. So it is possible that your DQN which performs "better" is just a red herring where Q learnt is bogus and over-optimistic. So when DDQN produces lower numbers it only looks "worse" but in fact is closer to realistic solution.. Just a thought..
Comments.UserDisplayName: Boppity Bop
Tags: <reinforcement-learning><q-learning><dqn><deep-rl><double-dqn>
Title: Can DQN perform better than Double DQN?
UserDisplayName: Angelo
ThreadText: # Can DQN perform better than Double DQN?
  
  (Asked by: username_0 on 2019-04-08)
  
  I'm training both DQN and double DQN in the same environment, but DQN performs significantly better than double DQN. As I've seen [in the double DQN paper](https://arxiv.org/pdf/1509.06461.pdf), double DQN should perform better than DQN. Am I doing something wrong or is it possible?
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl, double-dqn
  
  > **Comments:**
  > **username_1**: Hello. I am facing the same issue. What explanation finally did you give ?  
  > **username_2**: I have the same result, DQN performs better than Double DQN. I m try to understand why.  
  > **username_3**: DDQN was designed to address Q-function overestimation. So it is possible that your DQN which performs "better" is just a red herring where Q learnt is bogus and over-optimistic. So when DDQN produces lower numbers it only looks "worse" but in fact is closer to realistic solution.. Just a thought..  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2019-04-09. Score: 2 | Accepted answer
  
  There is no thorough proof, theoretical or experimental that Double DQN is better then vanilla DQN. There are a lot of different tasks, paper and later experiments only explore some of them. What practitioner can take out of it is that *on some tasks* DDQN is better. That's the essence of Deep Mind's "Rainbow" approach - drop a lot of different methods into bucket and take best results.
  
  > **Comments:**
  > **username_1**: Can you give some references where DQN outperforms DoubleDQN?  
  > **username_1**: I have a problem where my DQN is better than DoubleDQN and I cannot give a valid explanation on why.  
  
  ---
  
  ### Answer by username_5 on 2019-04-09. Score: 1
  
  That may happen when the value of the state is bad. You can find the example and explain about that in the link below.
  
  See this:<https://medium.freecodecamp.org/improvements-in-deep-q-learning-dueling-double-dqn-prioritized-experience-replay-and-fixed-58b130cc5682>
  
  > **Comments:**
  > **username_0**: What do you mean by "the value of the state is bad"? As I'm using an OpenAI-Gym environment, the value of the state is just the observation that I'm getting from it.  
  > **username_5**: @username_0  you can read the blog above to understand the answer . As you calculate all actions at one state but all these actions  do not affect the environment in a relevant way.  
  > **username_5**: @username_0 In most states, the choice of action has no effect.  
  > **username_6**: We prefer answers that contain the necessary info to answer the question, rather than answers that just link to another external resource, so I suggest that you provide the necessary info from that article in your answer.  
  > **username_1**: In your link, it says that in DuelingDQN session, not in DoubleDQN as asked.  
  > **username_5**: @username_1 It has both of them and you can see this: https://medium.com/@parsa_h_m/deep-reinforcement-learning-dqn-double-dqn-dueling-dqn-noisy-dqn-and-dqn-with-prioritized-551f621a9823  
  
  ---
  
  