Answers.Body: <p>This expression: <span class="math-container">$|\mathcal{A}(s)|$</span> means</p>
  <ul>
  <li><p><span class="math-container">$|\quad|$</span> the size of</p>
  </li>
  <li><p><span class="math-container">$\mathcal{A}(s)$</span> the set of actions in state <span class="math-container">$s$</span></p>
  </li>
  </ul>
  <p>or more simply the number of actions allowed in the state.</p>
  <p>This makes sense in the given formula because <span class="math-container">$\frac{\epsilon}{|\mathcal{A}(s)|}$</span> is then the probability of taking each exploratory action in an <span class="math-container">$\epsilon$</span>-greedy policy. The overall expression is the expected return when following that policy, summing expected results from the exploratory and greedy action.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-10T15:56:16.587Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118285/discussion-on-answer-by-neil-slater-what-does-the-term-mathcalas-mean-i).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-14T20:35:00.197Z
Answers.LastActivityDate: 2020-07-14T20:44:55.347Z
Answers.LastEditDate: 2020-07-14T20:44:55.347Z
Answers.UserDisplayName: Neil Slater
Body: <p>I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the <span class="math-container">$|A(s)|$</span> mean. I guess <span class="math-container">$A$</span> is the action set but I'm not sure about that notation:</p>
  <p><span class="math-container">$$\frac{\varepsilon}{|\mathcal{A}(s)|} \sum_{a} Q^{\pi}(s, a)+(1-\varepsilon) \max _{a} Q^{\pi}(s, a)$$</span></p>
  <p><a href="http://www.incompleteideas.net/book/first/ebook/node54.html" rel="nofollow noreferrer">Here</a> is the source of the formula.</p>
  <p>I also want to clarify that I understand the idea behind the <span class="math-container">$\epsilon$</span>-greedy approach and the motivation behind the on-policy methods. I just had a problem understanding this notation (and also some other minor things). The author there omitted some stuff, so I feel like there was a continuity jump, which is why I didn't get the notation, etc. I'd be more than glad if I can be pointed towards a better source where this is detailed.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-14T20:34:53.303Z
Comments.Text: From the pseudo code, it is pretty clear that $A(s)$ refers to the set of all possible actions, since in step *c)* the algorithm iterates through all actions ($a$) (taken from that set). That it is about the actions becomes apparent from the use of $a$.
Comments.UserDisplayName: Daniel B.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-14T20:44:04.983Z
Comments.Text: Yes I realized that I was talking more about the notation $|A(s)|$ but I get it now. Thanks.
Comments.UserDisplayName: Metrician
Tags: <reinforcement-learning><monte-carlo-methods><notation><on-policy-methods><epsilon-greedy-policy>
Title: What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?
UserDisplayName: Metrician
ThreadText: # What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?
  
  (Asked by: username_0 on 2020-07-14)
  
  I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that notation:
  
  $$\frac{\varepsilon}{|\mathcal{A}(s)|} \sum\_{a} Q^{\pi}(s, a)+(1-\varepsilon) \max \_{a} Q^{\pi}(s, a)$$
  
  [Here](http://www.incompleteideas.net/book/first/ebook/node54.html) is the source of the formula.
  
  I also want to clarify that I understand the idea behind the $\epsilon$-greedy approach and the motivation behind the on-policy methods. I just had a problem understanding this notation (and also some other minor things). The author there omitted some stuff, so I feel like there was a continuity jump, which is why I didn't get the notation, etc. I'd be more than glad if I can be pointed towards a better source where this is detailed.
  
  **Tags:** reinforcement-learning, monte-carlo-methods, notation, on-policy-methods, epsilon-greedy-policy
  > **Comments:**
  > **username_1**: From the pseudo code, it is pretty clear that $A(s)$ refers to the set of all possible actions, since in step *c)* the algorithm iterates through all actions ($a$) (taken from that set). That it is about the actions becomes apparent from the use of $a$.  
  > **username_0**: Yes I realized that I was talking more about the notation $|A(s)|$ but I get it now. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-14. Score: 6 | Accepted answer
  
  This expression: $|\mathcal{A}(s)|$ means
  
  * $|\quad|$ the size of
  * $\mathcal{A}(s)$ the set of actions in state $s$
  
  or more simply the number of actions allowed in the state.
  
  This makes sense in the given formula because $\frac{\epsilon}{|\mathcal{A}(s)|}$ is then the probability of taking each exploratory action in an $\epsilon$-greedy policy. The overall expression is the expected return when following that policy, summing expected results from the exploratory and greedy action.
  
  > **Comments:**
  > **username_3**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118285/discussion-on-answer-by-neil-slater-what-does-the-term-mathcalas-mean-i).  
  
  ---
  
  