<p>This expression: <span class="math-container">$|\mathcal{A}(s)|$</span> means</p>
<ul>
<li><p><span class="math-container">$|\quad|$</span> the size of</p>
</li>
<li><p><span class="math-container">$\mathcal{A}(s)$</span> the set of actions in state <span class="math-container">$s$</span></p>
</li>
</ul>
<p>or more simply the number of actions allowed in the state.</p>
<p>This makes sense in the given formula because <span class="math-container">$\frac{\epsilon}{|\mathcal{A}(s)|}$</span> is then the probability of taking each exploratory action in an <span class="math-container">$\epsilon$</span>-greedy policy. The overall expression is the expected return when following that policy, summing expected results from the exploratory and greedy action.</p>

CC BY-SA 4.0
2021-01-10T15:56:16.587Z
Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118285/discussion-on-answer-by-neil-slater-what-does-the-term-mathcalas-mean-i).
nbro
CC BY-SA 4.0
2020-07-14T20:35:00.197Z
2020-07-14T20:44:55.347Z
2020-07-14T20:44:55.347Z
Neil Slater
<p>I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the <span class="math-container">$|A(s)|$</span> mean. I guess <span class="math-container">$A$</span> is the action set but I'm not sure about that notation:</p>
<p><span class="math-container">$$\frac{\varepsilon}{|\mathcal{A}(s)|} \sum_{a} Q^{\pi}(s, a)+(1-\varepsilon) \max _{a} Q^{\pi}(s, a)$$</span></p>
<p><a href="http://www.incompleteideas.net/book/first/ebook/node54.html" rel="nofollow noreferrer">Here</a> is the source of the formula.</p>
<p>I also want to clarify that I understand the idea behind the <span class="math-container">$\epsilon$</span>-greedy approach and the motivation behind the on-policy methods. I just had a problem understanding this notation (and also some other minor things). The author there omitted some stuff, so I feel like there was a continuity jump, which is why I didn't get the notation, etc. I'd be more than glad if I can be pointed towards a better source where this is detailed.</p>

CC BY-SA 4.0
2020-07-14T20:34:53.303Z
From the pseudo code, it is pretty clear that $A(s)$ refers to the set of all possible actions, since in step *c)* the algorithm iterates through all actions ($a$) (taken from that set). That it is about the actions becomes apparent from the use of $a$.
Daniel B.
CC BY-SA 4.0
2020-07-14T20:44:04.983Z
Yes I realized that I was talking more about the notation $|A(s)|$ but I get it now. Thanks.
Metrician
<reinforcement-learning><monte-carlo-methods><notation><on-policy-methods><epsilon-greedy-policy>
What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?
Metrician
# What does the term $|\mathcal{A}(s)|$ mean in the $\epsilon$-greedy policy?

(Asked by: username_0 on 2020-07-14)

I've been looking online for a while for a source that explains these computations but I can't find anywhere what does the $|A(s)|$ mean. I guess $A$ is the action set but I'm not sure about that notation:

$$\frac{\varepsilon}{|\mathcal{A}(s)|} \sum\_{a} Q^{\pi}(s, a)+(1-\varepsilon) \max \_{a} Q^{\pi}(s, a)$$

[Here](http://www.incompleteideas.net/book/first/ebook/node54.html) is the source of the formula.

I also want to clarify that I understand the idea behind the $\epsilon$-greedy approach and the motivation behind the on-policy methods. I just had a problem understanding this notation (and also some other minor things). The author there omitted some stuff, so I feel like there was a continuity jump, which is why I didn't get the notation, etc. I'd be more than glad if I can be pointed towards a better source where this is detailed.

**Tags:** reinforcement-learning, monte-carlo-methods, notation, on-policy-methods, epsilon-greedy-policy

> **Comments:**
> **username_1**: From the pseudo code, it is pretty clear that $A(s)$ refers to the set of all possible actions, since in step *c)* the algorithm iterates through all actions ($a$) (taken from that set). That it is about the actions becomes apparent from the use of $a$.  
> **username_0**: Yes I realized that I was talking more about the notation $|A(s)|$ but I get it now. Thanks.  

---

## 1 answers

### Answer by username_2 on 2020-07-14. Score: 6 | Accepted answer

This expression: $|\mathcal{A}(s)|$ means

* $|\quad|$ the size of
* $\mathcal{A}(s)$ the set of actions in state $s$

or more simply the number of actions allowed in the state.

This makes sense in the given formula because $\frac{\epsilon}{|\mathcal{A}(s)|}$ is then the probability of taking each exploratory action in an $\epsilon$-greedy policy. The overall expression is the expected return when following that policy, summing expected results from the exploratory and greedy action.

> **Comments:**
> **username_3**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118285/discussion-on-answer-by-neil-slater-what-does-the-term-mathcalas-mean-i).  

---

