<p>Regarding your first question, <span class="math-container">$$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$</span> so recovering the value function from Q really depends on what policy <span class="math-container">$\pi$</span> you are using. Hence, you can't really recover the value function <span class="math-container">$V(s)$</span> from the <span class="math-container">$Q(s,a)$</span> values without knowing your policy distribution for state <span class="math-container">$s$</span>.</p>
<p>However, you can recover <span class="math-container">$Q^{\pi}(s,a)$</span> values if we know <span class="math-container">$V^{\pi}(s)$</span> and <span class="math-container">$A^{\pi}(s,a)$</span>. This is because
<span class="math-container">$$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$</span></p>
<p>by definition of advantage. And this is why you need 2 heads to recover the <span class="math-container">$Q$</span> values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover <span class="math-container">$Q^{\pi}(s,a)$</span> values due to &quot;identifability&quot; issue and the fact that both <span class="math-container">$V^{\pi}(s)$</span> and <span class="math-container">$Q^{\pi}(s,a)$</span> are only estimates.</p>
<p>Regarding your second question, I believe the author's appllied the Duelling architecture on <a href="https://arxiv.org/abs/1509.06461" rel="nofollow noreferrer">Double Deep Q Networks</a>, which is an improvement over the single DQN used by <a href="https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf" rel="nofollow noreferrer">Minh et al</a> in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.</p>

CC BY-SA 4.0
2021-03-23T13:12:34.333Z
Thanks for the answer. About the target network, the double DQN paper uses a target network the same as DQN does (see the original paper of Double DQN or Appendix A of the Dueling DQN paper). But, dueling DQN uses a SARSA update to get the target value. They have mentioned that in Section 4.1. Still, not sure why they have not used target network in Dueling DQN.
Afshin Oroojlooy
CC BY-SA 4.0
2021-03-23T14:31:29.667Z
I see what you mean. I am not very sure why they did not use a target network. It could be because that they are using the SARSA algorithm rather than Q-learning one. I think you can search for Deep SARSA methods to see whether the choice of network is similar.
calveeen
CC BY-SA 4.0
2021-03-26T15:15:04.497Z
@Media, Thanks for the reply. I noticed that they have used the DDQN framework for the Atari experiments, but was not sure which network is their proposal.
Afshin Oroojlooy
CC BY-SA 4.0
2021-03-23T04:19:18.260Z
2021-03-23T04:19:18.260Z
calveeen
<p>I have two questions on the Dueling DQN paper. <strong>First</strong>, I have an issue on understanding the identifiability that Dueling DQN paper mentions:</p>
<p><a href="https://i.sstatic.net/QDXyK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QDXyK.png" alt="enter image description here" /></a></p>
<p>Here is my question: If we have given Q-values <span class="math-container">$Q(s, a; \theta)$</span> for all actions, I assume we can get value for state <span class="math-container">$s$</span> by:</p>
<p><span class="math-container">$$V(s) = \frac {1} {|Q|} \sum_{a \in \mathcal{Q}} Q(s, a; \theta)$$</span>
and the advantage by:
<span class="math-container">$$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$</span></p>
<p>in which <span class="math-container">$\mathcal{A}(s)$</span> is the action space for state <span class="math-container">$s$</span>. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?</p>
<p>and then obtain Q-value using</p>
<p><span class="math-container">$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$</span></p>
<p>or
<span class="math-container">$$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$</span></p>
<p>Am I missing something?</p>
<p>My <strong>second</strong> question is why Dueling DQN does not use the target network as it is used in the DQN paper?</p>

CC BY-SA 4.0
2023-01-28T16:31:44.990Z
Please, next time, ask only one question per post.
nbro
<reinforcement-learning><dqn><value-based-methods><d3qn>
Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?
Afshin Oroojlooy
# Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?

(Asked by: username_0 on 2021-03-22)

I have two questions on the Dueling DQN paper. **First**, I have an issue on understanding the identifiability that Dueling DQN paper mentions:

[![enter image description here](https://i.sstatic.net/QDXyK.png)](https://i.sstatic.net/QDXyK.png)

Here is my question: If we have given Q-values $Q(s, a; \theta)$ for all actions, I assume we can get value for state $s$ by:

$$V(s) = \frac {1} {|Q|} \sum\_{a \in \mathcal{Q}} Q(s, a; \theta)$$
and the advantage by:
$$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$

in which $\mathcal{A}(s)$ is the action space for state $s$. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?

and then obtain Q-value using

$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max\_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$

or
$$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum\_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$

Am I missing something?

My **second** question is why Dueling DQN does not use the target network as it is used in the DQN paper?

**Tags:** reinforcement-learning, dqn, value-based-methods, d3qn

> **Comments:**
> **username_1**: Please, next time, ask only one question per post.  

---

## 1 answers

### Answer by username_2 on 2021-03-23. Score: 1 | Accepted answer

Regarding your first question, $$V^{\pi}(s) = \sum\_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$.

However, you can recover $Q^{\pi}(s,a)$ values if we know $V^{\pi}(s)$ and $A^{\pi}(s,a)$. This is because
$$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$

by definition of advantage. And this is why you need 2 heads to recover the $Q$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $Q^{\pi}(s,a)$ values due to "identifability" issue and the fact that both $V^{\pi}(s)$ and $Q^{\pi}(s,a)$ are only estimates.

Regarding your second question, I believe the author's appllied the Duelling architecture on [Double Deep Q Networks](https://arxiv.org/abs/1509.06461), which is an improvement over the single DQN used by [Minh et al](https://www.cs.toronto.edu/%7Evmnih/docs/dqn.pdf) in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.

> **Comments:**
> **username_0**: Thanks for the answer. About the target network, the double DQN paper uses a target network the same as DQN does (see the original paper of Double DQN or Appendix A of the Dueling DQN paper). But, dueling DQN uses a SARSA update to get the target value. They have mentioned that in Section 4.1. Still, not sure why they have not used target network in Dueling DQN.  
> **username_2**: I see what you mean. I am not very sure why they did not use a target network. It could be because that they are using the SARSA algorithm rather than Q-learning one. I think you can search for Deep SARSA methods to see whether the choice of network is similar.  
> **username_0**: @Media, Thanks for the reply. I noticed that they have used the DDQN framework for the Atari experiments, but was not sure which network is their proposal.  

---

