Answers.Body: <p>What you are referring to as the situation where </p>
  
  <blockquote>
    <p>some indexes are not available</p>
  </blockquote>
  
  <p>is simply the situation where some actions are not available/valid in some state. So, yes, the <span class="math-container">${\arg \max }$</span> will be calculated based only on the available actions in that state. More formally, 
  <span class="math-container">$$\underset{a \in \mathcal{A}(s)}{\arg \max } \, Q(s, a)$$</span> </p>
  
  <p>where <span class="math-container">$Q(s,a)$</span> has been initialized for all <span class="math-container">$s \in \mathcal{S}^{+}$</span> and <span class="math-container">$a \in \mathcal{A}(s)$</span> and <span class="math-container">$\mathcal{A}(s)$</span> is defined as the set of all actions available in state <span class="math-container">$s$</span>.
  See Sutton and Barto's <a href="http://incompleteideas.net/book/RLbook2020.pdf" rel="nofollow noreferrer">Intro to RL book</a>, chapter 6 (the part on Q-learning).</p>
  
  <p>In the same book (chapter 3), the authors state that: </p>
  
  <blockquote>
    <p><strong>To simplify notation</strong>, we sometimes assume the special case in which
    the action set is the same in all states and write it simply as <span class="math-container">$\mathcal{A}$</span>.</p>
  </blockquote>
  
  <p>Therefore, in many sources, you may see the <span class="math-container">$\arg \max$</span> expressed as <span class="math-container">$\underset{a \in \mathcal{A}} {\arg \max}$</span> or even <span class="math-container">$\underset{a} {\arg \max}$</span>. It's implicit that it is taken with respect to the set of actions <span class="math-container">$\mathcal{A}(s)$</span> available in state <span class="math-container">$s$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-03T22:13:00.043Z
Answers.LastActivityDate: 2020-06-03T22:13:00.043Z
Answers.UserDisplayName: user5093249
Body: <p>To update the Q table Q-learning takes the arg max of the Q values - the state, value mappings. </p>
  
  <p>For example, in tic tac toe the state <code>XOX OXO -X-</code> contains two available positions, each marked by the <code>-</code> character. In order to evaluate the arg max should <code>temporal difference</code> calculate the arg max of just the available positions? </p>
  
  <p>For the state <code>XOX OXO -X-</code> the arg max should be taken at index 6 and 8 ? (assuming zero indexing) If not then how should the arg max indexes be updated? The indexes <code>0,1,2 3,4,5 7</code> can be used as they are already been taken by X or O so should not be evaluated for their value ? This also means that indexes which are not available will not have their Q value updated, will this break the Q-learning procedure ?</p>
  
Tags: <reinforcement-learning><temporal-difference-methods>
Title: What is correct update when the some indexes are not available?
UserDisplayName: blue-sky
ThreadText: # What is correct update when the some indexes are not available?
  
  (Asked by: username_0 on 2020-06-02)
  
  To update the Q table Q-learning takes the arg max of the Q values - the state, value mappings.
  
  For example, in tic tac toe the state `XOX OXO -X-` contains two available positions, each marked by the `-` character. In order to evaluate the arg max should `temporal difference` calculate the arg max of just the available positions?
  
  For the state `XOX OXO -X-` the arg max should be taken at index 6 and 8 ? (assuming zero indexing) If not then how should the arg max indexes be updated? The indexes `0,1,2 3,4,5 7` can be used as they are already been taken by X or O so should not be evaluated for their value ? This also means that indexes which are not available will not have their Q value updated, will this break the Q-learning procedure ?
  
  **Tags:** reinforcement-learning, temporal-difference-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-03. Score: 3 | Accepted answer
  
  What you are referring to as the situation where
  
  > some indexes are not available
  
  is simply the situation where some actions are not available/valid in some state. So, yes, the ${\arg \max }$ will be calculated based only on the available actions in that state. More formally,
  $$\underset{a \in \mathcal{A}(s)}{\arg \max } \, Q(s, a)$$
  
  where $Q(s,a)$ has been initialized for all $s \in \mathcal{S}^{+}$ and $a \in \mathcal{A}(s)$ and $\mathcal{A}(s)$ is defined as the set of all actions available in state $s$.
  See Sutton and Barto's [Intro to RL book](http://incompleteideas.net/book/RLbook2020.pdf), chapter 6 (the part on Q-learning).
  
  In the same book (chapter 3), the authors state that:
  
  > **To simplify notation**, we sometimes assume the special case in which
  > the action set is the same in all states and write it simply as $\mathcal{A}$.
  
  Therefore, in many sources, you may see the $\arg \max$ expressed as $\underset{a \in \mathcal{A}} {\arg \max}$ or even $\underset{a} {\arg \max}$. It's implicit that it is taken with respect to the set of actions $\mathcal{A}(s)$ available in state $s$.
  
  
  ---
  
  