Answers.Body: <p>Regarding your first question, <span class="math-container">$$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$</span> so recovering the value function from Q really depends on what policy <span class="math-container">$\pi$</span> you are using. Hence, you can't really recover the value function <span class="math-container">$V(s)$</span> from the <span class="math-container">$Q(s,a)$</span> values without knowing your policy distribution for state <span class="math-container">$s$</span>.</p>
  <p>However, you can recover <span class="math-container">$Q^{\pi}(s,a)$</span> values if we know <span class="math-container">$V^{\pi}(s)$</span> and <span class="math-container">$A^{\pi}(s,a)$</span>. This is because
  <span class="math-container">$$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$</span></p>
  <p>by definition of advantage. And this is why you need 2 heads to recover the <span class="math-container">$Q$</span> values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover <span class="math-container">$Q^{\pi}(s,a)$</span> values due to &quot;identifability&quot; issue and the fact that both <span class="math-container">$V^{\pi}(s)$</span> and <span class="math-container">$Q^{\pi}(s,a)$</span> are only estimates.</p>
  <p>Regarding your second question, I believe the author's appllied the Duelling architecture on <a href="https://arxiv.org/abs/1509.06461" rel="nofollow noreferrer">Double Deep Q Networks</a>, which is an improvement over the single DQN used by <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">Minh et al</a> in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T13:12:34.333Z
Answers.Text: Thanks for the answer. About the target network, the double DQN paper uses a target network the same as DQN does (see the original paper of Double DQN or Appendix A of the Dueling DQN paper). But, dueling DQN uses a SARSA update to get the target value. They have mentioned that in Section 4.1. Still, not sure why they have not used target network in Dueling DQN.
Answers.UserDisplayName: Afshin Oroojlooy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T14:31:29.667Z
Answers.Text: I see what you mean. I am not very sure why they did not use a target network. It could be because that they are using the SARSA algorithm rather than Q-learning one. I think you can search for Deep SARSA methods to see whether the choice of network is similar.
Answers.UserDisplayName: calveeen
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T15:15:04.497Z
Answers.Text: @Media, Thanks for the reply. I noticed that they have used the DDQN framework for the Atari experiments, but was not sure which network is their proposal.
Answers.UserDisplayName: Afshin Oroojlooy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-23T04:19:18.260Z
Answers.LastActivityDate: 2021-03-23T04:19:18.260Z
Answers.UserDisplayName: calveeen
Body: <p>I have two questions on the Dueling DQN paper. <strong>First</strong>, I have an issue on understanding the identifiability that Dueling DQN paper mentions:</p>
  <p><a href="https://i.sstatic.net/QDXyK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QDXyK.png" alt="enter image description here" /></a></p>
  <p>Here is my question: If we have given Q-values <span class="math-container">$Q(s, a; \theta)$</span> for all actions, I assume we can get value for state <span class="math-container">$s$</span> by:</p>
  <p><span class="math-container">$$V(s) = \frac {1} {|Q|} \sum_{a \in \mathcal{Q}} Q(s, a; \theta)$$</span>
  and the advantage by:
  <span class="math-container">$$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$</span></p>
  <p>in which <span class="math-container">$\mathcal{A}(s)$</span> is the action space for state <span class="math-container">$s$</span>. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?</p>
  <p>and then obtain Q-value using</p>
  <p><span class="math-container">$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$</span></p>
  <p>or
  <span class="math-container">$$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$</span></p>
  <p>Am I missing something?</p>
  <p>My <strong>second</strong> question is why Dueling DQN does not use the target network as it is used in the DQN paper?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-28T16:31:44.990Z
Comments.Text: Please, next time, ask only one question per post.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><dqn><value-based-methods><d3qn>
Title: Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?
UserDisplayName: Afshin Oroojlooy
ThreadText: # Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?
  
  (Asked by: username_0 on 2021-03-22)
  
  I have two questions on the Dueling DQN paper. **First**, I have an issue on understanding the identifiability that Dueling DQN paper mentions:
  
  [![enter image description here](https://i.sstatic.net/QDXyK.png)](https://i.sstatic.net/QDXyK.png)
  
  Here is my question: If we have given Q-values $Q(s, a; \theta)$ for all actions, I assume we can get value for state $s$ by:
  
  $$V(s) = \frac {1} {|Q|} \sum\_{a \in \mathcal{Q}} Q(s, a; \theta)$$
  and the advantage by:
  $$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$
  
  in which $\mathcal{A}(s)$ is the action space for state $s$. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?
  
  and then obtain Q-value using
  
  $$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max\_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$
  
  or
  $$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum\_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$
  
  Am I missing something?
  
  My **second** question is why Dueling DQN does not use the target network as it is used in the DQN paper?
  
  **Tags:** reinforcement-learning, dqn, value-based-methods, d3qn
  > **Comments:**
  > **username_1**: Please, next time, ask only one question per post.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-23. Score: 1 | Accepted answer
  
  Regarding your first question, $$V^{\pi}(s) = \sum\_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$.
  
  However, you can recover $Q^{\pi}(s,a)$ values if we know $V^{\pi}(s)$ and $A^{\pi}(s,a)$. This is because
  $$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$
  
  by definition of advantage. And this is why you need 2 heads to recover the $Q$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $Q^{\pi}(s,a)$ values due to "identifability" issue and the fact that both $V^{\pi}(s)$ and $Q^{\pi}(s,a)$ are only estimates.
  
  Regarding your second question, I believe the author's appllied the Duelling architecture on [Double Deep Q Networks](https://arxiv.org/abs/1509.06461), which is an improvement over the single DQN used by [Minh et al](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf) in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.
  
  > **Comments:**
  > **username_0**: Thanks for the answer. About the target network, the double DQN paper uses a target network the same as DQN does (see the original paper of Double DQN or Appendix A of the Dueling DQN paper). But, dueling DQN uses a SARSA update to get the target value. They have mentioned that in Section 4.1. Still, not sure why they have not used target network in Dueling DQN.  
  > **username_2**: I see what you mean. I am not very sure why they did not use a target network. It could be because that they are using the SARSA algorithm rather than Q-learning one. I think you can search for Deep SARSA methods to see whether the choice of network is similar.  
  > **username_0**: @Media, Thanks for the reply. I noticed that they have used the DDQN framework for the Atari experiments, but was not sure which network is their proposal.  
  
  ---
  
  