Answers.Body: <p>This is considered in the original paper, but is rejected due to training instability. To quote:</p>
  <blockquote>
  <p>[The average-advantage formulation] increases the stability of the optimization...the advantages only need to change as fast as the mean, instead of having to compensate any change to the optimal action's advantage in [the formulation where $Q(s, a) is normalized with the best action's advantage].</p>
  </blockquote>
  <p>Furthermore, the authors attempt a softmax version, but they find it doesn't perform that differently than the average-advantage version. I couldn't find any additional experiments using this, but it may be worth trying if it is absolutely necessary to obtain an accurate <span class="math-container">$V(s)$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-02T19:29:31.313Z
Answers.LastActivityDate: 2021-07-02T19:29:31.313Z
Answers.UserDisplayName: Daniel M.
Body: <p>In a Dueling DQN agent (<a href="https://arxiv.org/pdf/1511.06581.pdf" rel="nofollow noreferrer">Wang et al.</a>), the Q function is decomposed as</p>
  <p><span class="math-container">$$
  Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum_{a'\in \mathcal{A}}A(s, a')
  $$</span></p>
  <p>representing the value of the state, plus the advantage of the action, minus the average advantage of all actions available in that state.</p>
  <p>However, this formulation means that the value of the given state is skewed, even after subtracting out the mean advantage of all actions available. Why isn't it set up so that the advantage of the best action is 0 (with the other actions' advantages being negative), thus leading to a more accurate <span class="math-container">$V(s)$</span>?</p>
  
Tags: <reinforcement-learning><dqn>
Title: In a DDQN architecture, why is the value of a state assumed to be the average of the Q values of the actions?
UserDisplayName: Daniel M.
ThreadText: # In a DDQN architecture, why is the value of a state assumed to be the average of the Q values of the actions?
  
  (Asked by: username_0 on 2021-07-02)
  
  In a Dueling DQN agent ([Wang et al.](https://arxiv.org/pdf/1511.06581.pdf)), the Q function is decomposed as
  
  $$
  Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum\_{a'\in \mathcal{A}}A(s, a')
  $$
  
  representing the value of the state, plus the advantage of the action, minus the average advantage of all actions available in that state.
  
  However, this formulation means that the value of the given state is skewed, even after subtracting out the mean advantage of all actions available. Why isn't it set up so that the advantage of the best action is 0 (with the other actions' advantages being negative), thus leading to a more accurate $V(s)$?
  
  **Tags:** reinforcement-learning, dqn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-07-02. Score: 1
  
  This is considered in the original paper, but is rejected due to training instability. To quote:
  
  > [The average-advantage formulation] increases the stability of the optimization...the advantages only need to change as fast as the mean, instead of having to compensate any change to the optimal action's advantage in [the formulation where $Q(s, a) is normalized with the best action's advantage].
  
  Furthermore, the authors attempt a softmax version, but they find it doesn't perform that differently than the average-advantage version. I couldn't find any additional experiments using this, but it may be worth trying if it is absolutely necessary to obtain an accurate $V(s)$.
  
  
  ---
  
  