Answers.Body: <p>I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method is more concerned with overestimation induced by function approximation error rather than stochasticity in the environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-17T14:16:57.920Z
Answers.Text: I highly recommend, if you haven't done so already, to take a look at the author's implementation of TD3, https://github.com/sfujim/TD3. It's extremely clean and simple, so it's very easy to read. I know two of the authors personally and they've told me they can code up the algorithm and train it on HalfCheetah or something *during* a presentation, so like really quickly.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-27T03:58:43.457Z
Answers.LastActivityDate: 2019-03-27T03:58:43.457Z
Answers.UserDisplayName: Luke Guye
Body: <p><a href="https://ai.stackexchange.com/questions/11418/why-q2-is-a-more-or-less-independant-estimate-in-twin-delayed-ddpg-td3">Twin Delayed Deep Deterministic (TD3)</a> policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained on different samples. In double DQN, I understand that target Q and current Q are relatively independent because their parameters are quite different. </p>
  
  <p>But in TD3, Q1 and Q2 are trained on exactly the same target. If their parameters are initialized the same, there will be no difference in their output and the algorithm will be equal to DQN. The only source of independence/difference of Q2 to Q1 I can tell is the randomness in the initialization of their parameters. But with training on the same target, I thought this independence will become smaller and smaller as they converge to the same target values. So I don't quite understand why TD3 works in combating overestimation in Q-learning.</p>
  
Tags: <reinforcement-learning><q-learning><dqn><deep-rl><ddpg>
Title: Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?
UserDisplayName: Luke Guye
ThreadText: # Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?
  
  (Asked by: username_0 on 2019-03-24)
  
  [Twin Delayed Deep Deterministic (TD3)](https://ai.stackexchange.com/questions/11418/why-q2-is-a-more-or-less-independant-estimate-in-twin-delayed-ddpg-td3) policy gradient is inspired by both double Q-learning and double DQN. In double Q-learning, I understand that Q1 and Q2 are independent because they are trained on different samples. In double DQN, I understand that target Q and current Q are relatively independent because their parameters are quite different.
  
  But in TD3, Q1 and Q2 are trained on exactly the same target. If their parameters are initialized the same, there will be no difference in their output and the algorithm will be equal to DQN. The only source of independence/difference of Q2 to Q1 I can tell is the randomness in the initialization of their parameters. But with training on the same target, I thought this independence will become smaller and smaller as they converge to the same target values. So I don't quite understand why TD3 works in combating overestimation in Q-learning.
  
  **Tags:** reinforcement-learning, q-learning, dqn, deep-rl, ddpg
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-03-27. Score: 2
  
  I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method is more concerned with overestimation induced by function approximation error rather than stochasticity in the environment.
  
  > **Comments:**
  > **username_1**: I highly recommend, if you haven't done so already, to take a look at the author's implementation of TD3, https://github.com/sfujim/TD3. It's extremely clean and simple, so it's very easy to read. I know two of the authors personally and they've told me they can code up the algorithm and train it on HalfCheetah or something *during* a presentation, so like really quickly.  
  
  ---
  
  