Answers.Body: <p>Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.</p>
  
  <p><a href="https://i.sstatic.net/WKrmS.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WKrmS.png" alt=""></a></p>
  
  <blockquote>
    <p>From <a href="https://arxiv.org/pdf/1511.06581" rel="nofollow noreferrer">Wang et al., 2016</a>, Dueling Network Architectures for Deep Reinforcement Learning</p>
  </blockquote>
  
  <p>On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result in the identifiability issue. That is the network might simply learn <span class="math-container">$V(s)=0$</span>, <span class="math-container">$A(s,a)=Q(s,a)$</span> for every state.</p>
  
  <p>To tackle with this issue, we should force an additional constraint on the advantage estimate. We can simply use the equation below as mentioned in the paper, that is, normalize the advantages across actions before combining with the state value.</p>
  
  <p><span class="math-container">$$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+(A(s,a;\theta,\alpha)-\frac{1}{|A|}\sum\limits_{a'}A(s,a';\theta,\alpha))$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T12:52:52.343Z
Answers.LastActivityDate: 2020-04-14T12:59:10.343Z
Answers.LastEditDate: 2020-04-14T12:59:10.343Z
Answers.UserDisplayName: J3soon
Body: <p>I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs</p>
  
  <ol>
  <li><p>Advantage: how good it is to be in a particular state <span class="math-container">$s$</span></p></li>
  <li><p>Value: the advantage of choosing a particular action <span class="math-container">$a$</span></p></li>
  </ol>
  
  <p>We can make these two outputs into Q values (reward for choosing particular action <span class="math-container">$a$</span> when in state <span class="math-container">$s$</span>) by adding them together.</p>
  
  <p>However, in a DQN, we get Q values from the single output layer of the network. </p>
  
  <p>Now, suppose that I use the same DQN model with the very same weights in my input and hidden layers and changing the output layer which gives us Q values to advantage and value outputs. <em>Then, during training, if I add them together, will it give me the same Q value for a particular state, supposing all the parameters of both my algorithms are the same except for the output layers?</em></p>
  
Tags: <machine-learning><reinforcement-learning><q-learning><dqn><deep-rl>
Title: Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?
UserDisplayName: Raj
ThreadText: # Are Q values estimated from a DQN different from a duelling DQN with the same number of layers and filters?
  
  (Asked by: username_0 on 2020-04-13)
  
  I am confused about the Q values of a duelling deep Q network (DQN). As far as I know, duelling DQNs have 2 outputs
  
  1. Advantage: how good it is to be in a particular state $s$
  2. Value: the advantage of choosing a particular action $a$
  
  We can make these two outputs into Q values (reward for choosing particular action $a$ when in state $s$) by adding them together.
  
  However, in a DQN, we get Q values from the single output layer of the network.
  
  Now, suppose that I use the same DQN model with the very same weights in my input and hidden layers and changing the output layer which gives us Q values to advantage and value outputs. *Then, during training, if I add them together, will it give me the same Q value for a particular state, supposing all the parameters of both my algorithms are the same except for the output layers?*
  
  **Tags:** machine-learning, reinforcement-learning, q-learning, dqn, deep-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-14. Score: 4
  
  Dueling-DQN has different network architecture comparing to vanilla DQN, so I don't think your version will work as well as the Dueling architecture.
  
  [![](https://i.sstatic.net/WKrmS.png)](https://i.sstatic.net/WKrmS.png)
  
  > From [Wang et al., 2016](https://arxiv.org/pdf/1511.06581), Dueling Network Architectures for Deep Reinforcement Learning
  
  On the other hand, since we only have the target Q-value, separating the Q-value into state value and advantage result in the identifiability issue. That is the network might simply learn $V(s)=0$, $A(s,a)=Q(s,a)$ for every state.
  
  To tackle with this issue, we should force an additional constraint on the advantage estimate. We can simply use the equation below as mentioned in the paper, that is, normalize the advantages across actions before combining with the state value.
  
  $$Q(s,a;\theta,\alpha,\beta)=V(s;\theta,\beta)+(A(s,a;\theta,\alpha)-\frac{1}{|A|}\sum\limits\_{a'}A(s,a';\theta,\alpha))$$
  
  
  ---
  
  