Answers.Body: <h3>For behaviour policy</h3>
  <p>It's kind of a free choice since the main rules for the behaviour policy are that</p>
  <ul>
  <li><p>It should cover all possible choices. This is handled outside of the Q function by adding some randomness</p>
  </li>
  <li><p>For learning efficiency it should be <em>close</em> to the target policy. This is a loose, nice to have, requirement.</p>
  </li>
  </ul>
  <p>So any of the current Q being learned should do as the basis of <span class="math-container">$\epsilon$</span>-greedy behaviour policy. Using average or alternating between them also works, but there is no requirement to match to the one being updated or the one being used to generate the maximising action for the update.</p>
  <h3>For final/output policy</h3>
  <p>The estimators are expected to converge to approximations of the true optimal action value function, with only minor differences, so you could use either at the end.</p>
  <p>Using both and averaging them might offer a small boost to accuracy (similar to aggregating estimators in other scenarios).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-15T12:28:23.247Z
Answers.LastActivityDate: 2023-02-16T07:29:48.160Z
Answers.LastEditDate: 2023-02-16T07:29:48.160Z
Answers.UserDisplayName: Neil Slater
Body: <p>I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy. Or whether we choose one over the other. It seems to me that when playing around with it typically one of the q functions would outperform the sum or average of both q functions.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-16T07:22:28.077Z
Comments.Text: Do you mean behaviour policy, or output policy for evaluation/final use?
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><q-learning>
Title: Which Q function do we use to make our policy when using double Q learning?
UserDisplayName: IloveR
ThreadText: # Which Q function do we use to make our policy when using double Q learning?
  
  (Asked by: username_0 on 2023-02-15)
  
  I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy. Or whether we choose one over the other. It seems to me that when playing around with it typically one of the q functions would outperform the sum or average of both q functions.
  
  **Tags:** reinforcement-learning, q-learning
  > **Comments:**
  > **username_1**: Do you mean behaviour policy, or output policy for evaluation/final use?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-15. Score: 3 | Accepted answer
  
  ### For behaviour policy
  
  It's kind of a free choice since the main rules for the behaviour policy are that
  
  * It should cover all possible choices. This is handled outside of the Q function by adding some randomness
  * For learning efficiency it should be *close* to the target policy. This is a loose, nice to have, requirement.
  
  So any of the current Q being learned should do as the basis of $\epsilon$-greedy behaviour policy. Using average or alternating between them also works, but there is no requirement to match to the one being updated or the one being used to generate the maximising action for the update.
  
  ### For final/output policy
  
  The estimators are expected to converge to approximations of the true optimal action value function, with only minor differences, so you could use either at the end.
  
  Using both and averaging them might offer a small boost to accuracy (similar to aggregating estimators in other scenarios).
  
  
  ---
  
  